FYI these screen proportions are slightly wrong. the 3ds top screen is 400x240 (ratio: 5/3 ~ 1.667) and the bottom screen is 320x240 (ratio: 4/3 ~ 1.333) so to best fit the monitor, which from your numbers looks to be about 1000px high and maintain the aspect ratio you'll need to do a little math. The aspect ratios that you have in your custom layout is just a little off (1.660 and 1.323)
The following numbers should better maintain the aspect ratio of the screens while still fitting the ultrawide monitor
Top Screen: (1667x1000 Ratio: 1.667)
Bottom Screen: (Resolution: 893x670 Ratio: 1.3328)
I'm the guy who added the original support for screen layouts and this is the reason why i'm still not sold on the current implementation for custom layouts. I love the idea, but having 8 boxes that take a lot of tweaking to get the layout to look good is just not user friendly (and the boxes don't respect aspect ratio for better and for worse!). When it comes to features in the GUI, we try really hard to make sure the features are user friendly and I'd prefer a different solution than the current setup. But i've been too lazy to actually work on the solution i have in mind ....