More specifically, I will be using libp2p-webrtc-star for my application. The regular WebRTC uses SCTP for its data channel and it allows you to specify whether it will use in-order delivery and whether data transfer is reliable. But the documentation for webrtc-star has no mention of things like these?
libp2p-webrtc-star is currently the most reliable transport for libp2p in the browser. It comes with built in discovery.
Behind the scenes it uses https://github.com/feross/simple-peer . So, it is perhaps better to check what is configurable from there. We should definitely have this in the Readme. On dial you can provide simple peer options https://github.com/libp2p/js-libp2p-webrtc-star/blob/v0.20.4/src/index.js#L99 . Same for the listener.
Let me know if you have any issues trying it out.