Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add Socket connection? #22

Open
zoakes opened this issue Nov 24, 2019 · 27 comments
Open

Add Socket connection? #22

zoakes opened this issue Nov 24, 2019 · 27 comments
Labels
Enhancement

Comments

@zoakes
Copy link

zoakes commented Nov 24, 2019

I've been thinking about adding a socket connection class to a generic FIX IP, was curious if there's anything in beta for this? If not, I'm happy to fork and contribute what I can to it if I have some success -- I was going to connect to IB.

Let me know your thoughts!

Simplefix is INCREDIBLE, so simple -- exactly what I was looking for -- FIX formatting without the overhead mess of QuickFix -- just needs a Socket initiator : )

Zach
@da4089
Copy link
Owner

da4089 commented Nov 25, 2019

Hi Zach,

At one time I was fairly opposed to this idea. I think it's a slippery slope from adding some sort of socket support, and then maybe some sort of message validation, and suddenly we're back where QuickFIX is.

But I'm maybe less against it than I was, mostly because I think a lot of people end up having to write it themselves, and while that's not terribly hard, it does seem like maybe an off-the-shelf way to do it would be better.

So. This is maybe a little unfair, but I'd kinda be interested in seeing what you come up with, without really promising to merge the PR if I get cold feet. But ... since you're writing it anyway, and since you're kinda volunteering, I'm absolutely happy to look at a PR and work with you on merging it if it seems promising?

@zoakes
Copy link
Author

zoakes commented Nov 25, 2019 via email

@zoakes
Copy link
Author

zoakes commented Nov 25, 2019 via email

@zoakes
Copy link
Author

zoakes commented Nov 26, 2019

I added some stuff that most people would probably want -- like heartbeat, logging, etc. I think for now I'm done, this worked for me today in some preliminary testing, so I'm pretty pleased hah. I expected this fix engine to take like.. weeks-months, so thank you for saving me that : )
End goal is Cython / C++ engine, so I'm going to get started on optimizing tomorrow.

I'm happy to work with you on the PR (pull request?) -- I'm not super familiar with all the github ins and outs, but I'm a solid developer. If it's beginning to feel a bit like QF, no need to merge -- I'm indifferent, really I'm happy I have that makes sense and works well in a few hundred lines. Let me know !

Thank you again for this incredible module.

Zach

@zoakes
Copy link
Author

zoakes commented Dec 20, 2019

David,

I finished a really clean Python FIX client with simplefix -- I'll attach the current version here. I finally finished up integrating / testing with Interactive Brokers IB CTCI FIX API, they are incredibly particular about what's used, what isn't, what's in header, etc -- but it's ready now!
If you'd like to use it, or pull it -- you're welcome to.
I currently have it running the heartbeat as a Daemon thread, but I have a few versions of heartbeat that can be used -- this has been the most consistent.

https://github.com/zoakes/sFIX-IB/blob/master/base_hb_v2.0.py

I didn't add any message validation, I'm not quite sure how to aside from just using a listener -- which sounds bloated. If it is something you're interested in I can give it a try though.

@mecanda12
Copy link

mecanda12 commented Apr 4, 2023 •
edited
Loading

@da4089 @zoakes Hey David,

If Zach, code is worth it, is any possibility to merge it? I see his code help connect to the backend engine.

@da4089
Copy link
Owner

da4089 commented Apr 5, 2023

I will definitely have a look at this. I'm still unsure how I'd like to integrate it, but at a minimum, it could be in some sort of example code directory. The next couple of weeks are a little busy, but I should get a chance to look at it after that.

Thank you for this!

@mecanda12
Copy link

Thanks, David :)

@zoakes
Copy link
Author

zoakes commented Apr 5, 2023 via email

@da4089
Copy link
Owner

da4089 commented Apr 5, 2023

For better or worse, there's not a lot of usage of the new session layer stuff that I've seen: most FIX is still using the classic session protocol and some variant of FIX4.x.

But yes, a production quality FIX engine is a heap of work. In most cases, you'd probably be better off starting with the Python QuickFIX or WTFIX if that's what you need.

That said, something that's reasonably resilient and suitable for running regression tests, etc, isn't too much effort.

@mecanda12
Copy link

@da4089 I am looking for a solution for a Non-production environment. Most a Test environment.
My question is the same how can I make a network connection using simplefix, QuickFIX, or WTFIX?

@zoakes
Copy link
Author

zoakes commented Apr 5, 2023 via email

@zoakes
Copy link
Author

zoakes commented Apr 5, 2023 via email

@mecanda12
Copy link

mecanda12 commented Apr 5, 2023 •
edited
Loading

@zoakes To make a connection with the backend engine, a socket can be used to communicate with the bundle.

Ip address, Port, Target & Sender comp Id, Target Sub ID, User address and password.

The above information can be used to make a network connection.

@zoakes
Copy link
Author

zoakes commented Apr 5, 2023 via email

@mecanda12
Copy link

@zoakes I do not think regression testing needs that much $$.
Keep in mind I am talking about internal-level testing. Which will be happening on the same network.

Do you have a solution for that?

@zoakes
Copy link
Author

zoakes commented May 14, 2023 •
edited
Loading

Sure, you can mock a FIX endpoint. Quickfix has one ready to go, though you'll need to hard code various behaviors into it to sort of 'mock' the behavior you expect wrt resets, rejections, sessions, etc.
good example here would be some exchanges seem to elide the 'Ack' with the 'Fill' for marketable orders.
Others don't.
Some (most?) also permit cancellation with a ClOrdId (pre ack), while others require the venue given OrderId (post ack).

There's lots of nuance, even in mocking expected venue behavior / state.

To connect to a real endpoint (for any form of real testing), I think you will need to pass certification & fully integrate for the given endpoint. In modern fix, that means a multicast VPN, and multiple sessions for various 'cert specific' products.

That's always been the case for me.
My SFIX-IB was basically a hacked solution to pass certification with IB, but eventually I used QuickFix to manage the overhead of things like sequence reset, in production.
I think I used it in prod for weeks before realizing it was easier to use basic api, and was of no benefit otherwise.

Real fix is complicated, and the certifications require handling a lot of edge cases.
CME has steps like... send 60,000 limit orders, we'll fill / reject them randomly, then cancel any remaining.
Now add a disconnect + reset / reconcile halfway through, etc.
IB's was exponentially easier, but it's also very manual whereas cme is automated certification, just a testing suite.

I don't really see the point of testing the robustness of a mocked connection, personally, but that's just me.
These multicast, distributed systems are not trivial to test, I find it more efficient to write larger tests that cover as much as possible.
Again -- that's purely my preference.

@Umair-A
Copy link

Umair-A commented Feb 23, 2024

Hello, there!
As, I'm new to the FIX protocol I do not understand where I can start and how I can use this if zoakes or anyone else can give me any suggestions or documents to understand this it will be very helpful for me the real fix documentation is really overwhelming and going over my head.

@da4089
Copy link
Owner

da4089 commented Feb 23, 2024

Hello, there! As, I'm new to the FIX protocol I do not understand where I can start and how I can use this if zoakes or anyone else can give me any suggestions or documents to understand this it will be very helpful for me the real fix documentation is really overwhelming and going over my head.

What are you trying to achieve?

@Umair-A
Copy link

Umair-A commented Feb 23, 2024 •
edited
Loading

My end goal is to create a connection with the mt5 server or you can say a liquidity provider. For it, I must have a clear understanding of FIX API and how it will work.

@zoakes
Copy link
Author

zoakes commented Feb 23, 2024

Not following... I don't think MT5 has FIX, I don't know why they would.
Fix would be more for connecting directly to venues -- cme, nyse, Nasdaq, bzx, edga, edgx etc.

Some brokers provide endpoints that can then route to many venues -- trading technologies, interactive brokers.

I don't know where a trading platform would come in.
FIX is just a messaging protocol.

I see no mention of mt5 offering a fix endpoint, don't see what purpose it would serve their users.

@Umair-A
Copy link

Umair-A commented Feb 23, 2024

Exactly, I will get an endpoint from the broker. I wanted to get market feed data and other order sending on it etc. If you could please look into this repository it is what exactly I want but it is an old repository which is not working now.
https://github.com/rinleit/quickfix-python-samples

@zoakes
Copy link
Author

zoakes commented Feb 23, 2024 •
edited
Loading

Python quickfix works fine.
I've used it for drop copy from dozens of exchanges, and used it to hit TT, IB.

FIX is complicated, and very not generic -- if it doesn't work you're likely not meeting their spec, or aren't permissioned to hit the endpoint.
It's whatever the provider (acceptor, in QF parlance) says it is.

Most brokers have a UAT environment, and onboarding process to use it that you must pass.
Contact them, is my suggestion -- they'll provide all this.

Of note -- this is not quickfix.
Please direct quickfix related issues to quickfix.

This repo is more 'I want to build my own fix client, to replace Quickfix, and just don't want to do the message parsing / creation myself'

Also of note -- 9/10 times I hear this, someone is interested in latency. Fix is NOT inherently faster, and unless in newer binary formats it's most likely going to offer SLOWER execution speed, particularly with new unrefined implementations in things like quickfix, w basic network setup etc.
So much of latency will be network, hardware, and has nothing to do w connectivity method -- but there's not much that's naturally fast about the enormous parsing overhead of FIX.

Like... making a fix parser as fast as possible, will get you a couple us faster, but your network jitter alone is likely in the ms.... what was the purpose?

If this is a latency play -- this won't fix anything. Been there, done that.

@Umair-A
Copy link

Umair-A commented Feb 26, 2024

Thanks for the kind response. The problem with QuickFix Python it is not working and giving this error while installing it on a Windows machine
image
And I got your point thanks

@Umair-A
Copy link

Umair-A commented Feb 26, 2024

And If you can please give me steps on how can I achieve a connection with any server using quickfix or fix API. Please mention some steps or material if it is possible.

@zoakes
Copy link
Author

zoakes commented Feb 26, 2024 via email •
edited
Loading

@da4089
Copy link
Owner

da4089 commented Feb 27, 2024

And If you can please give me steps on how can I achieve a connection with any server using quickfix or fix API. Please mention some steps or material if it is possible.

@Umair-A it sounds like your best option would be to ask the QuickFIX folks for some help, probably a pointer to an example or tutorial.

The steps involved in making a FIX connection include

  • Getting the required "Rules of Engagement" document
  • Getting the connectivity details
  • Establishing and maintaining TCP connectivity
  • Establishing and maintaining FIX connectivity
    • Including implementing FIX recovery
  • Implementing the messages you need for your application (quotes, orders, symbol lists, positions, etc)

In some of these steps, there are decisions to make about how you'd like to implement them. Network programming in Python has a number of possible approaches; message and session state persistence can also be done various ways.

And if your ultimate goal is integration with MT5, that is itself a pretty large project (and completely unrelated to SimpleFIX).

SimpleFIX would, at most, only be a very small part of all this. I'm happy to help you with that bit of the project, but the rest of it is your own responsibility.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Enhancement
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@da4089 @mecanda12 @zoakes @Umair-A