Tuesday September 29th, 2020



Script Troubles

So I left my script running over night and unfortunately I got logged out at some point. Bummed about this. I think there might be a cookie rotation they do to really ensure. I might have to respond properly to these requests to update the cookie I have stored. May need to dig deeper into the login flow instead of trying to hijack an existing session.

Digging through the Capital One Login I see a few things. It looks like when the form is submitted they generate an encrypted username and password. There is also another Base64 encoded payload with some fakable params. I am not sure all the things they are generating in that.

I dug through the code just a little bit and saw some public keys that looked like they are used to encrypt the username and password, however it didn’t look super deterministic. It seems there might be some other injection there to help rotate the encryption or something.. I am really not sure here. I notice they are also generating JS which gets sent to the client, I’m wondering if this has something to do with it.

Public Keys

-----END PUBLIC KEY-----
-----END PUBLIC KEY-----

For now I am going to let another experiment run (logging what cookies it wants me to set). In the meantime will start to refactor and see what I can do.

Also if I do try to go for this login method I will probably do this with a headless browser or something, then hijack the cookie and use it.

So over 4 hours still logged in. This is good, but need to catch any failures.

7 hours and still in…

Okay finally I got logged out, right around 8 hours it seems. I think because of this I will try to log in headlessly then use the cookie. This also will give me more control so I don’t have to constantly ping the server. Anyway this was a fun experiment.


The refactor into a CapitalOne client was super smooth. Stupid easy and makes the code readable.

While refactoring more things I realized that it might be good to separate all this into a library. I think I will call this hyperverse. This is something that will need more thought, so I am continuing to work on the existing code base then decide how it should be split later.

Hmmm so even that is going to need more thinking. I do not want to over-abstract right now. I can think of a few ways but we just need to start writing the code. For now I have just set up the structs that will ultimately be sent out in the GraphQL queries.


Speeding up Daily Generation

Yesterday I had a lot of problems with generating this page. Well mostly that it was really slow and painful. Today I am starting to write some SW which should ease the pain before I start figuring out really what I want.

Issues with the old workflow

Previously the workflow was

  1. Write .md
  2. Generate .html
  3. Git commit (on local)
  4. Git pull (on webserver)

We basically want to automate all these steps except 3. 3 is basically me ‘publishing’ so I am fine to keep this manual. I might later automate this, but for now I will keep it under my control. (the automation would be on save go git commit)

Automation Time!

Fixing pain between markdown and html

First thing is when I save a .md file to trigger html generation so I don’t have to think. I normally would do this with inotifywatch on linux, but I write on my Mac. Looks like there is a similar tool called fswatch. I will use this instead

brew install fswatch

After installation I needed to think about what I want the trigger to be. I am thinking any .md file which is modified.

This ended up being a bit more of a pain than I expected. fswatch doesn’t quite behave as I would expect it to. Specifically -i & -I is for inclusive regex. However by default EVERYTHING(!!) is included. This means you filter out everything with -e .* then include back what you want with -i *.md

Ultimately my command/script ended up as:

fswatch -xnr -e ".*" -i ".md" ../serve/day | while read file event; do
  python3 gen_html.py $file

Basically what this does is look for any .md files that have changed in the dir ../serve/day recursively. If a file has changed, then gen_html.py will be ran with the filename.

gen_html.py is a little script that does a few things

  1. Gets the title of the markdown file
  2. Generates html using pandoc
  3. Inserts this generated html into a template
  4. Writes out the final html output

lol this is basically dead simple static site generation

Anyway this helps to alleviate the pain between 1 & 2. This is great. The number of button clicks was too damn high

Fixing pain between commit (on local) and pull (on webserver)

Okay so now that we have this html we want to make sure the webserver gets it. (i guess i could host on github, but whatever for now we will do this. maybe later move to github just so other people can deploy easier. at least this is a good local webserver 😋. some of the reason I am not sure about moving it to github is that I might not use git forever. right now it is convienient, but not sure if that’s really what I want.)

To do this I am going to use git hooks. When I git push something I am going to send a ping to the webserver to tell it to pull down the latest commit.

This took me a bit longer than I expected. Basically wasn’t even sure how the git hook gets ran. After a little bit of digging it’s quite clear. It runs from .git/hooks. There are a variety of hooks listed here. Basically what you do is name a file the same name as the hook. Make sure it’s executable and git will try to execute it. I ran into a problem with my SHEBANG for python.. After fixing that it ran without issue.

Now when I git push the webserver will be updated in 30sec. Again there were probably tons of approaches to do this, but it was fun to play with some git hooks, even if not super portable (ie .git folder stays only on the local)


Is it possible to have a spreadsheet view of a GraphQL DB?

What I mean is it would be pretty convienient to have some times. Then apply formulas directly on top. But also have it linked to all my other data. Even for some data input this would be suuuuuper nice. Like right now I have to manually input a bunch of data for my investments.

query GetPlaidInstitutions {
  financial_institutions(where: {financial_plaid_items: {id: {_is_null: false}}}) {


Man after watching that debate I am not even sure what I want to do tomorrow. I almost want to go through the whole thing again and pick out what I thought was good and bad on both sides. What I agree and disagree with in whole. Something in simple language. I guess we’ll see. I do still want to get transaction data into the DB (even if not Capital One, well, perhaps I will do that too. But plaid can go ahead for now).

I also want to work more on the Supreme Court case writing too.

Quick list of some things I could do:

Okay so tomorrow: