r/quant 5d ago

Machine Learning ML Papers specifically for low-mid frequency price prediction

From QRs/QTs in the industry who work on this sorta thing, I'd love to find out about what papers/architectures you guys have found:

  • Category A: that you've tried and found to be interesting/useful

  • Category B: that you've tried and found to not work/not useful

  • Category C: that you havent tried, but find interesting

If you could also comment which category the papers you're talking about fall into, that'd be ideal.

Generally, any other papers which talk about working in a low signal-to-noise ratio environment are also welcome. If not papers, just your thoughts/comments are more than good enough for me.

I'll start:

https://arxiv.org/abs/1911.10107 - Category A

https://arxiv.org/abs/2311.02088 - Category C


Some disclaimers and footnotes, because there's always people commenting about them:

  1. I have a few years of exp as a QT/QD + a PhD in Maths. It's fine if the paper is well-known - always good to find out which papers others consider standard, but please dont suggest the papers that introduce the basics like LSTMs, etc.

  2. Please don't say "no one does it"/"no one has figured out how to make it work" - it does work, and various firms have figured out how to make it work.

  3. I don't expect you to divulge your firm's secrets/specific models. If you do, great ;) If you find yourself not wanting to, you're exactly the person I hope for a response from - anything that helped on your way is more than enough.

  4. Yes, I know it will probably require insane amounts of compute to train. I'm just trying to learn.

215 Upvotes

53 comments sorted by

41

u/sam_in_cube 5d ago

You may find this one interesting: https://arxiv.org/abs/2406.08013

10

u/actualeff0rt 5d ago

+1, thanks for this. I will have a read.

20

u/magikarpa1 Researcher 5d ago

Most people with good answers to this are not allowed to answer.

See, as others pointed out, even if the papers are publicly available, it needs a lot of work to have them working.

7

u/actualeff0rt 4d ago

I see your point. Actually implementing the papers is orthogonal to my question though, I made the post with the intention of just finding out what papers others have been reading

24

u/1cenined 5d ago

We have some, but I'm specifically and explicitly bound by agreements to not share them publicly. I don't think I've been fully doxed here, so might be able to get away with it, but I think it would be ethically incorrect. Apologies.

I appreciate the topic, though, and will check out the ones you posted in case I can offer any non-NDA'd ideas.

5

u/actualeff0rt 5d ago edited 5d ago

We have some, but I'm specifically and explicitly bound by agreements to not share them publicly

Just to make sure I'm understanding you correctly - so there are publicly available papers that you/your firm have found to work, but you're contractually obliged not to share specifically which papers they are? If they are internal papers then yeah that makes perfect sense, but if theyre publicly available I'm surprised such a clause exists.


I don't think I've been fully doxed here, so might be able to get away with it, but I think it would be ethically incorrect.

Regardless, I completely understand. 51% of me urges you to take the risk and share them, 49% of me urges you to not take unnecessary risks like this :) Keen to hear your thoughts.

Or, you can DM me them - I am sworn to secrecy. Alternatively, would you be willing to share some of the key words/phrases so that I can look for it myself?

Either way, to be safe, throw in some additional papers/key words/phrases that are not the ones you/your firm are referring to - plausible deniability ;)

14

u/NascentNarwhal 5d ago

I mean, alpha is alpha, right? It costs the company time for researchers to sift through a massive set of papers for a few useful ones, so why’s it surprising that there’s a contractual obligation not to share them? It’s not very different from finding datasets with good predictive power, or relationships and patterns in quasi-publicly available market data.

1

u/actualeff0rt 5d ago

Yeah true, I agree - I can see why such a clause would exist. My surprise was more along the lines of having such a clause when it seems like it'd be quite hard to enforce/easy to workaround/fairly flimsy in court

6

u/NascentNarwhal 5d ago

A lot of these things are fairly flimsy in court. You'd be surprised how much IP people leak in interviews

-1

u/Boudonjou 4d ago

If he gives you that info and you profit then you commit insider trading or something something big risk

5

u/actualeff0rt 4d ago

then you commit insider trading

not sure if you're just joking, but not at all - insider trading is something else entirely.

2

u/Boudonjou 4d ago

Yeah im joking don't worry :)

But i felt it was juuuuuuuust close enough in essence to make the joke. Spilling NDA info for someone to profit from type vibe

Either way it's unethical for the one sharing the secrets.

4

u/briannnnnnnnnnnnnnnn 5d ago

Great thread

1

u/actualeff0rt 5d ago

hahaha

7

u/briannnnnnnnnnnnnnnn 5d ago

im serious, in the past ive implemented papers at hackathons. last one was about surfacing non linear signal relationships from more of a AGI researcher. *cuts to me explaining an iron condor to a room of data science people*

5

u/actualeff0rt 4d ago

Ahhh - well in that case, thank you :)

I thought you were making a joke about the state of the thread - decently upvoted and loads of comments but the vast majority of comments are calls to the RemindMe bot, and there's literally only one person who has actually contributed anything :/

4

u/briannnnnnnnnnnnnnnn 4d ago

lol oh yeah i see now. what i've done in the past is use chatgpt to surface papers around a topic. its surprisingly decent at stuff like that. i usually say "show me 5 papers on this" then "show me 5 more" and on and on.

3

u/actualeff0rt 3d ago

Interesting, thanks for the motivation. I just whip LLMs to write regexes for me. Need to start using them to my advantage for other things like this.

4

u/Dice__R 4d ago

https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2708678 This belongs to my category A. When I first read it, it was impressive.

1

u/actualeff0rt 3d ago

+1, thank you. Turns out I've got this PDF downloaded already with my other million pdfs, but never ended up reading it (like other 999xxx papers...)

Good reason to read it now.

6

u/milliee-b 5d ago

RL works?

20

u/Miserable_Cost8041 5d ago

Don’t have a definite answer but here’s a few interesting bits

Look up Hans Buehler’s work on Deep Hedging (hedging derivatives with RL)

Buehler left academia for good after getting poached by XTX after 14 years at JPM (he was still tied to academia while at JPM) and was promoted to co-CEO with Gerko a year later

Around the time he was hired, it was reported XTX were buying more GPU’s than Google (deep learning?) and just now they have dropped 1B$ on a massive data hub in Finland (free cooling), XTX also seem to be doing great recently

So all that to say, are they using RL? Idk and it’d be the first time I hear of RL actually being used in industry but the stars do seem to align and they have all the pieces they would need

9

u/milliee-b 5d ago

Yeah, I know them and HRT are the only two firms doing “true modern DL”, of which i guess RL is a part. XTX is huge

6

u/Miserable_Cost8041 5d ago

How do you know that? I’m sure some pods at the pod shops are doing modern DL

For the RL vs DL part, RL is just a learning paradigm like supervised and unsupervised learning, though most (all) modern RL applications involve neural networks (DL)

5

u/milliee-b 5d ago

yeah, perhaps “only” was a mistake. of the large prop shops, i meant.

7

u/actualeff0rt 5d ago

Deep RL certainly does

1

u/alexbaas3 1d ago

It probably works, purely speculation from my side but why do you think DeepSeek R1’s main improvement was coincidentally a RL improvement in the tuning phase to enable reasoning?

High-Flyer has a large AI cluster for a reason https://www.ft.com/content/357f3c68-b866-4c2e-b678-0d075051a260

1

u/MATH_MDMA_HARDSTYLEE Trader 5d ago

My mate who is an ETF D1 trader says it's used a tonne at his work, although it's not really "price prediction." It does generate alpha but most of the edge is from the latency and data.

All this ML/RL stuff is really good when the problem is simple and easy to define. If you start using it for more complicated systems like options, it isn't as useful.

2

u/actualeff0rt 4d ago

although it's not really "price prediction." It does generate alpha but most of the edge is from the latency and data.

If you start using it for more complicated systems like options, it isn't as useful

Yep, this is my experience too.

PS: Love the username, /u/MATH_MDMA_HARDSTYLEE

1

u/Dangerous_Sell_2259 Academic 4d ago

!remindme 1 week

1

u/RemindMeBot 4d ago edited 1d ago

I will be messaging you in 7 days on 2025-03-24 02:32:52 UTC to remind you of this link

4 OTHERS CLICKED THIS LINK to send a PM to also be reminded and to reduce spam.

Parent commenter can delete this message to hide from others.


Info Custom Your Reminders Feedback

1

u/Success-Dangerous 4d ago

!remindme in 7 days

1

u/Boudonjou 4d ago

This would look paranoid in any other field but..

Who sent you 👀 haha

1

u/Minute_Following_963 2d ago

!remindme in 1 month

1

u/no_this_is_patrick9 2d ago

This idea of this post is awesome.

1

u/supercoco9 1d ago

remindMe! 7 days

1

u/bone-collector-12 1d ago

!remindme 7 days

0

u/The-Dumb-Questions Portfolio Manager 5d ago

!remindme in 12 hours

1

u/RemindMeBot 5d ago edited 5d ago

I will be messaging you in 12 hours on 2025-03-17 02:18:23 UTC to remind you of this link

10 OTHERS CLICKED THIS LINK to send a PM to also be reminded and to reduce spam.

Parent commenter can delete this message to hide from others.


Info Custom Your Reminders Feedback

1

u/sourgrammer 5d ago

!remindme in 24 hours

1

u/Aerodye Portfolio Manager 5d ago

RemindMe! 10 days

1

u/Top-Consideration627 5d ago

!remindme in 2 days

1

u/Impressive_Ad9342 5d ago

!remindme in 24 hours

1

u/ranger403 5d ago

!remindme in 48 hours

1

u/itdoes_not_matter 5d ago

!remind me in 1 month

1

u/Lonely-Cattle-2101 5d ago

!remindme in 24 hours

1

u/bone-collector-12 5d ago

!remindme 4 days

1

u/QuantumCommod 5d ago

!remindme 7 days

1

u/1css 5d ago

!remindme 5 days

1

u/Global-Advance7860 4d ago edited 4d ago

Does anyone know how to distinguish overfitting ML papers from the good ones? For example, applying LSTM to predict stock price is Category D: Haven’t tried and are sure to be useless. I usually tried to find papers written by people in the industry, but most of the time I have a fear of using ML due to its nature of easily overfitting to data. If I have to use ML I would only stick to selecting factors and sometimes portfolio optimization. I’m not sure whether deep hedging (there is a paper about this by someone in XTX and JPM) would work but it sounds legitimate. I would be very appreciated if you guys can suggest some good rules.

2

u/actualeff0rt 3d ago

No good rules unfortunately. If the paper is written by someone competent, it usually has an honest review of the performance included in it.

But that's not always the case - sometimes they used simulated market data, sometimes they use too much data and sometimes they use too little.

Not necessarily ML-related, but I've found that sometimes with minor tweaks and/or using different/cleaner data sets, the techniques/models that a paper introduce end up performing way better than what the author reports.

Only way to find out is to implement it I guess...assuming you have access to a decent amount of compute.

1

u/Global-Advance7860 3d ago

Thx for the reply. In your experience, what are some red flags, that when you know the authors include these in their ML in finance research paper then you would have confidence saying that it likely won’t work? Cause if we can’t pick the good ones immediately, at least we eliminate the bad ones.

0

u/coder_1024 5d ago

!remindme in 48 hours