Cylindrical Panorama with fisheye camera (project mode)

VIOSO6 provides a very neat way to align panoramic projections on curved screens up to 360° using a single fisheye calibration kit. This article describes such procedure based on VIOSO 6 calibration, using the 3D mapping project mode.

This approach is suitable for contents that wrap around the screen/cylinder in an observer-fee manner (“like a canvas”) and therefore is useful for media servers, video players, GPU hooking (Windows desktop) and similar applications.

For rendering content based on realtime rendering 3D engines (simulators, etc.) the approach differs. Please consult our simulation team for more information about this kind of calibrations. This approach assumes that the screen geometry is cylindrical, which means that it has one radius and even height. For more complex screen geometries like cones, elliptical screens, etc. you need to create a custom 3D model including a proper UV map (even pixel distribution) that matches precisely the physical screen geometry.

1. Prerequisites

As usual when dealing with autoalignment, a proper projector lineup and projector configuration is the key for best image quality as well as for a precise alignment and robust recalibration.

Projectors

1. Avoid using ultra-short throw projectors or ultra-short throw lenses when projecting on curved screens. The distortion caused by such projectors as well as the defocus will degrade very noticeably both image quality itself (pixel size, focus, color representation) and ability to calibrate.

2. Make sure the projector images covers the screen completely from top to bottom

   

3. Avoid tiling projectors, make use of lens-shift instead.

4. Portrait mounted projectors are a good way to meet high resolution demands on large panoramic screens.

Camera

1. Choose our high resolution camera kit (CK_FDQ). Due to the peripherical scanning the high resolution sensor matters.

2. Make sure to mount the camera in the very center of the panoramic screen, either on the ceiling (ideal for fixed installations) or on the floor (for temporary camera placement).

3. Having something obstructing the camera view on the projection like scaffolding, handrails, the projectors itself etc. adds another challenge, so try to avoid.

4. By no means something must block the view on the overlaps.

5. Having control on the general lighting conditions is mandatory. Consider to execute the calibration only in pitch black condition (no ambient light) for robust results.

Considerations for seamless 360° projections and simulators

1. You cannot warp the calibrated result in a 2D way, so all image adjustments must be made in the 3D alignment. Make sure to have the proper experience dealing with this constraint before executing the calibration onsite.

2. The stitching line of the content must NOT happen in a projector overlap since it will create a small visible like. Please consider this when planning the projector layout.

Make sure all requirements are met, all projectors are prepared, the camera is properly mounted or placed in its final position before proceeding with the calibration.

Preparations onsite

1. Make sure the projectors are mounted properly and meet the requirements listed above.

2. Create a testpattern matching the aspect ratio of the screen. Important: Do not create a testpattern using the full resolution, as this will distort the output. Learn more about creating a proper testpattern here: Testpattern

3. Configure the camera to expose a resolution with 1:1 aspect ratio.

Example (Daheng cameras):

4. Have the camera properly mounted or placed. Make sure that nothing is obstructing the projector beams nor the view of the camera on the projection.

5. Make sure any ambient light is turned off before starting the scanning.

Calibration

3.1 Designer

Launch VIOSO 6 and run in Project Mode. Create a new project according to the designated playback solution. We will use Windows Desktop Warp as an example for this tutorial.

Set up the projection configuration accordingly:

1. Computer node:

   • Mosaic settings are detected automatically. Right-click on each display to access further features like output splitting

   • Projectors: Add all projectors and connect

2. Projectors

   • Add all projectors and connect them with the computer node

3. Camera

   • Set the resolution according to the camera

   • Set the camera to “Fisheye lens” and 185° opening Angle

   • If you know the position and rotation of the camera in relation to the screen, you can enter the values below “Pose Parameter”, otherwise keep them set to 0 since the parameters can be set in the “Add 3D” step later in this process

4. Surface

   • Use the built-in 3D generator for basic geometries (plane, cylinder, dome, etc.) Enter the radius and the image height of the cylinder. Enter the curvature in degree from the center of the screen. Example values:

     • 360° seamless cylinder: left=-180°, right=180°

     • 270 ° cylinder: left=-135°, right=135°

     • 180° cylinder: left=-90°, right=90°

   • Or load a 3D model with appropriate UV Map:

     • Normals need to point to the inside

     • UV map must cover the inside of the cylinder as well

5. Activate Parameter

   • Here you specify where to export to. Please consult the respective documentation for the target system (media server, realtime-engine, etc.) how to handle the export.

   • For applying to a NVIDIA Mosaic, there is not much to set apart from the “apply on start up” checkbox, which makes VIOSO 6 apply the warp&blend whenever started

3.2. Scanning Procedure

3.3.1. Pre-Scan Settings

Choose “Grid/Arbitrary”, since the other modes won’t work on fisheye cameras.

• Check “set display calibration mask” to create masks on projectors => strongly recommended

• Check “force new geometric scan” if this is the first run or you intend to do a full rescan

3.3.2. Camera finetuning & masking

The camera is started and you see the projected checkerboard pattern.

• Toggling crosshairs help you to align the camera evenly in the center (A)

• Double-check that the camera is set to the custom 1:1 resolution (B)

• Use the “Format” button to access the camera’s parameters. We recommend to set to these values as a good starting point

  • Acquisition control:

    • Exposure: 40.000 (sensitive, slow) – 80.000 (less sensitive, faster scans)

  • Analog Control:

    • Gamma: OFF

    • Sharpening: OFF (Important!)

After positioning and finetuning, create a mask to have the camera view limited to the visible area (=area of interest). It’s important to mask anything that could distract the scanning like reflective areas, light spots, etc.

A typical mask for a fisheye on a cylinder looks like this:

3.3.3. Projector masking

When activated, in the next step masks can be created and applied to each projector. This is important to avoid overshooting on floor and ceiling.

Create masks to limit the projection to the screen:

After repeating for each projector, the outcome should look like this:

3.3.3. Scan

Once the camera and all masks are set up, the scanning procedure will be straightforward: Make sure the camera sensitivity is properly set – if too bright adjust either the camera’s iris or dim the projector brightness down. The resulting image should look similar to this:

A preview of a dot pattern is generated. Choose a dot size as small as possible where you are still able to identify each dot clearly. If in doubt, start with dot size 1.2 for standard fisheye kits (2k x 2kresolution), and 1.0 for HQ fisheye kits (4k x 4k resolution).

The resulting picture should look like this:

Click on “Next” to proceed to the dot pattern and “Next” to initiate the scanning.

After each scan, the result of the scan is pictured by a sample output on the projector and an informative rendering on the screen.

We recommend to use the fisheye testing pattern ” Domegrid_4K-LoW_22.png” found in the current users’s Pictures folder or at C:\Program Files\VIOSO6\Footage\Dome

Check that the projector is completely covered, while the projector masking is applied. This results in a projector output like this:

Note: Don't mind if the displayed image looks flipped and not ready aligned - we're handling this later. The aim of this preview is to visualize the pure scan result.

If you see errors like wavy lines, black spots on the projector, edit the extrapolation parameters or repeat the scan.

Click “Next” to proceed with the other projectors, that you handle in the same way.

4. Align Camera and surface (3D Mapping)

After finishing the scan (everything is saved automatically in the folder “Calibration” of the current project directory), it’s about to align with the camera position with the screen position. Based on that information, the content is distributed equally and linear on the screen.

Edit the camera position either by moving and translating with the interactive tools (A) or by entering/editing the values directly (B). Make sure that the camera parameters are set (C).

This procecure requires some practise since it is highly handling the whole scenerey from the view of an fisheye lens. Learn more about how to handle this 3D mapping tool here.

As a result, the whole screen should be covered with the testing pattern, getting a fairly good geometric alignment (not 100% accuracy is required at this step unless you go for seamless 360° panoramas):

Remarks

• The content is a realtime-rendered texture based on the UV map of the model.

• Make sure that all the projection is covered with this testpattern. Leaving spots blank (black) will leave black spots for all subsequent steps. Therefore, we recommend to have a certain overshooting happening at this step, since it can be compensated the subsequent 2D Editing step.

• Try to align the lines with the screen geometry as good as it gets, but don’t strive to get a 100% result. There are some facts that will jeopardize this:

  • 3D-model and the screen don’t fit exactly

  • Camera distortion

Therefore, once you have a certain feeling to have the best possible alignment done (=no black parts, everything more or less centered), you are good to continue.

Note for seamless 360° projection and simulators:

For seamless 360° projections, every geometric editing must happen in this 3D alignment step. The additional warping of the subsequent step cannot apply for seamless 360° projections.

5. Edit (Warp & Blend)

The second step of editing happens after the conversion from 3D into the 2D space of the content.

Depending on the camera resolution and the number of projectors, this transformation can take several minutes. Make sure not to interrupt this transformation, which is executed automatically.

5.1. Warping

Click “Player” and add the testpattern generated in step 2.2 (Preparation) and click Play to display it on the projectors. It will fit to the screen according to the way the 3D alignment has been done before.

Use the warping grid to fine-align the testpattern according to the screen geometry. A typical warp might look like this:

Resulting output on the projection:

5.2.Edit Blending & projector balance

Refer to this article to learn more about how to optimize the softedge blending.

6. Using the calibration

Refer to the documentation for integrating the calibrated result depending on your use-case:

• Integrated Videoplayer

• Exaplay

• Desktop Embedding on NVIDIA Quadro GPUs

• 3rd party Media Playback Solutions

7. Recalibrate & Realign

7.1 Recalibrate

Just push the button “Recalibrate” and the whole scanning procedure will run through fully automated:

• Disable evtl. applied desktop warp (on NVIDIA Hooking)

• Launch the calibration based on all previously saved parameters.

• Add 3D alignment and coversions

• Perform the 2D warp and softedge blending

• Export (upading existing files)

• Re-apply desktop warp (on NVIDIA Hooking)

7.2 Re-align

If case of a camera movement, the calibration must be run through manually.

• First, check that the camera is fully working.

• Second, create a backup of the current project by duplicating the whole project folder.

• Then click “Calibration” and proceed to step 3.2.2 where the camera image is displayed. A typical misaligned camera image will look like this:

The camera mask (red parts) can act as a reference. When set tightly around the projection, the mask reveals the amount of offsetting. Use this for your favor and move the camera back into it’s position where it fits the mask as good as possible.

The image after realignment should look like this:

After realigning the camera, you run through the scanning procedure as outlined in 3.3.3. You should not change any parameter, as we assume that no other change than the camera position has been done.

After the scan, the 3D alignment needs to be double-checked, according to chapter 4. Here, depending on how precise you were able to push back the camera into it’s original position, only very few, deliberate modifications should be done. Concentrate mainly to not have blank parts on the screen. If the deviations are too heavy, don’t proceed but go back to the previous state and work on the alignment of the camera. The better the camera is physically realigned, the easier it will be to maintain the 3D alignment.

After converting, proceed with the warping according to chapter 5. Use the 2D warp to finetune the result, as it is likely that the previous mapping doesn’t fit exactly.

After that the realignment is done and everything can continue as before.