So I left my script running over night and unfortunately I got logged out at some point. Bummed about this. I think there might be a cookie rotation they do to really ensure. I might have to respond properly to these requests to update the cookie I have stored. May need to dig deeper into the login flow instead of trying to hijack an existing session.
Digging through the Capital One Login I see a few things. It looks like when the form is submitted they generate an encrypted username and password. There is also another Base64 encoded payload with some fakable params. I am not sure all the things they are generating in that.
I dug through the code just a little bit and saw some public keys that looked like they are used to encrypt the username and password, however it didn’t look super deterministic. It seems there might be some other injection there to help rotate the encryption or something.. I am really not sure here. I notice they are also generating JS which gets sent to the client, I’m wondering if this has something to do with it.
-----BEGIN PUBLIC KEY-----
MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAzIRgyU/g0W9Xiuv1ZUU8
8y/msQmyjTB4jaCDW5g6CxMQuiyavpPNpcqm3X2Cu6JV3XnvpzOplI0CmTHr3Wwz
HRtwc0xFiDUu5hqcI9bXlcI1wMpUqDemCOg4ya7v03ONFEqErXNuvPFebQ5bwTds
sOv619TNjP/oxUGrVXIHl7B8ai9o6/w2IF+OF/a3UpVCIjiqVSqizQXu/xdsSsqw
N+mBCR2vBGeJzabCJNmUE284/f15R08KWCyAr0QF1r8ih1MwqOVjFrcmoevWS09v
1y9tAB6ksnmL1511mewxpa3J6Nn9u4KvBYtfkwopy42HOlQvMEmJBBtleKYkE5u8
sQIDAQAB
-----END PUBLIC KEY----------BEGIN PUBLIC KEY-----
MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAiwcxZqZy5tgQiWuI6e0r
zoBelUXEuxtNLeCqFChS4peivGH0IP+QDnp2tOKQ2dA3sT8qu5YdOXhOc5RXC1Kq
P/Bbhvk0/R4m5UgPtq8YQcpbrC39GLyIWSEpGgDq7adjBa3cDoQdtkG3AfYVP6rK
FpKpMZZi0/MzUf+FMLnydqDez2pHAXzZTefq4OaUukBiKum764z7hEtNmxHQd4LN
hxcz6CHDlPyuWRYkDQt6S1iPCFnT0VZTjaEFTyUIMFDFFT1FhBVW+D9CIaLu/WAC
Az4QKEilsYZGOn7+vMhozKz2yzVFzRQI/r+WvnBzVFaY4BbbJd37dFzggpsUf+ze
9wIDAQAB
-----END PUBLIC KEY-----For now I am going to let another experiment run (logging what cookies it wants me to set). In the meantime will start to refactor and see what I can do.
Also if I do try to go for this login method I will probably do this with a headless browser or something, then hijack the cookie and use it.
So over 4 hours still logged in. This is good, but need to catch any failures.
7 hours and still in…
Okay finally I got logged out, right around 8 hours it seems. I think because of this I will try to log in headlessly then use the cookie. This also will give me more control so I don’t have to constantly ping the server. Anyway this was a fun experiment.
The refactor into a CapitalOne client was super smooth. Stupid easy and makes the code readable.
While refactoring more things I realized that it might be good to separate all this into a library. I think I will call this hyperverse. This is something that will need more thought, so I am continuing to work on the existing code base then decide how it should be split later.
Hmmm so even that is going to need more thinking. I do not want to over-abstract right now. I can think of a few ways but we just need to start writing the code. For now I have just set up the structs that will ultimately be sent out in the GraphQL queries.
Yesterday I had a lot of problems with generating this page. Well mostly that it was really slow and painful. Today I am starting to write some SW which should ease the pain before I start figuring out really what I want.
Previously the workflow was
We basically want to automate all these steps except 3. 3 is basically me ‘publishing’ so I am fine to keep this manual. I might later automate this, but for now I will keep it under my control. (the automation would be on save go git commit)
First thing is when I save a .md file to trigger html generation so I don’t have to think. I normally would do this with inotifywatch on linux, but I write on my Mac. Looks like there is a similar tool called fswatch. I will use this instead
brew install fswatch
After installation I needed to think about what I want the trigger to be. I am thinking any .md file which is modified.
This ended up being a bit more of a pain than I expected. fswatch doesn’t quite behave as I would expect it to. Specifically -i & -I is for inclusive regex. However by default EVERYTHING(!!) is included. This means you filter out everything with -e .* then include back what you want with -i *.md
Ultimately my command/script ended up as:
fswatch -xnr -e ".*" -i ".md" ../serve/day | while read file event; do
python3 gen_html.py $file
doneBasically what this does is look for any .md files that have changed in the dir ../serve/day recursively. If a file has changed, then gen_html.py will be ran with the filename.
gen_html.py is a little script that does a few things
title of the markdown filepandoclol this is basically dead simple static site generation
Anyway this helps to alleviate the pain between 1 & 2. This is great. The number of button clicks was too damn high
Okay so now that we have this html we want to make sure the webserver gets it. (i guess i could host on github, but whatever for now we will do this. maybe later move to github just so other people can deploy easier. at least this is a good local webserver 😋. some of the reason I am not sure about moving it to github is that I might not use git forever. right now it is convienient, but not sure if that’s really what I want.)
To do this I am going to use git hooks. When I git push something I am going to send a ping to the webserver to tell it to pull down the latest commit.
This took me a bit longer than I expected. Basically wasn’t even sure how the git hook gets ran. After a little bit of digging it’s quite clear. It runs from .git/hooks. There are a variety of hooks listed here. Basically what you do is name a file the same name as the hook. Make sure it’s executable and git will try to execute it. I ran into a problem with my SHEBANG for python.. After fixing that it ran without issue.
Now when I git push the webserver will be updated in 30sec. Again there were probably tons of approaches to do this, but it was fun to play with some git hooks, even if not super portable (ie .git folder stays only on the local)
What I mean is it would be pretty convienient to have some times. Then apply formulas directly on top. But also have it linked to all my other data. Even for some data input this would be suuuuuper nice. Like right now I have to manually input a bunch of data for my investments.
query GetPlaidInstitutions {
financial_institutions(where: {financial_plaid_items: {id: {_is_null: false}}}) {
id
logo
name
plaid_insitution_id
}
}Man after watching that debate I am not even sure what I want to do tomorrow. I almost want to go through the whole thing again and pick out what I thought was good and bad on both sides. What I agree and disagree with in whole. Something in simple language. I guess we’ll see. I do still want to get transaction data into the DB (even if not Capital One, well, perhaps I will do that too. But plaid can go ahead for now).
I also want to work more on the Supreme Court case writing too.
Quick list of some things I could do:
Okay so tomorrow: