Very Large Dataset, How to Combine into One Product?

Clinton Carman · Oct 19, 2018

Recently I completed a very large mapping project of a mine up in the mountains that was affected by a large wildfire. In total there was over 2,000 ft of elevation difference, spread across 3,500 acres. I broke the project into approximately 33 smaller flights using a Phantom 4 Pro v2.0, and used MapsMadeEasy for their Terrain Awareness feature. The flights were conducted at an altitude of 250 ft AGL which gave me a GSD of 0.8 in/px. Overall I've ended up with over 40,000 photos which are about 300GB of data.

I have processed each one of the flights using Pix4D Mapper and have those individual orthos, 3D meshes, point clouds, DSM's, etc. What I'd like to do now is combine everything into one single map so we can blow it up and give it to the client so they can see the whole picture. I'm fine keeping the smaller flights as high-resolution references, and having the combined document be lower resolution.

In the past I've successfully combined two maps in Pix4D Mapper desktop, but finding the 5-6 common points, labeling them identically, then reprocessing them took quite a while. To do that for all 33 maps would take me months.

Any ideas from you guys on how to successfully combine these into one final product?

TYIA

R Martin · Oct 19, 2018

Clinton Carman said:
Recently I completed a very large mapping project of a mine up in the mountains that was affected by a large wildfire. In total there was over 2,000 ft of elevation difference, spread across 3,500 acres. I broke the project into approximately 33 smaller flights using a Phantom 4 Pro v2.0, and used MapsMadeEasy for their Terrain Awareness feature. The flights were conducted at an altitude of 250 ft AGL which gave me a GSD of 0.8 in/px. Overall I've ended up with over 40,000 photos which are about 300GB of data.

I have processed each one of the flights using Pix4D Mapper and have those individual orthos, 3D meshes, point clouds, DSM's, etc. What I'd like to do now is combine everything into one single map so we can blow it up and give it to the client so they can see the whole picture. I'm fine keeping the smaller flights as high-resolution references, and having the combined document be lower resolution.

In the past I've successfully combined two maps in Pix4D Mapper desktop, but finding the 5-6 common points, labeling them identically, then reprocessing them took quite a while. To do that for all 33 maps would take me months.

Any ideas from you guys on how to successfully combine these into one final product?

Before you go down that rabbit hole you need to determine what the client wants and don't suggest a single file. Let them decide what they need. The hardware required to display a tif that size is going to beyond what most people possess. You can import the tifs into specialized software like ArcGIS and set up a map service and then break them into smaller individual tiles that are uniform. Of course the customer is going to have to have at least one seat on the service to use it.

Then again, the customer might be fine viewing the individual tif tiles you created as part of the initial processing.

Clinton Carman · Oct 19, 2018

R Martin said:
Before you go down that rabbit hole you need to determine what the client wants and don't suggest a single file. Let them decide what they need. The hardware required to display a tif that size is going to beyond what most people possess. You can import the tifs into specialized software like ArcGIS and set up a map service and then break them into smaller individual tiles that are uniform. Of course the customer is going to have to have at least one seat on the service to use it.

Then again, the customer might be fine viewing the individual tif tiles you created as part of the initial processing.

R Martin, thank you for the reply.

The customer does want a single map of the whole property. They want to be able to look at the whole property and get a scale of the damage, then if they need to look in an area/sector with more detail, they can load the individual map corresponding to that sector. Therefore, the map of the whole property doesn't have to have nearly the same resolution, have geo information like a GeoTIFF. It just needs to show the orthomosaic for the whole property. It could even be downsized to a JPEG to make it more user-friendly. They'd also like to do a printed image of the whole property that they can put up on their wall.

R Martin · Oct 19, 2018

Clinton Carman said:
R Martin, thank you for the reply.

The customer does want a single map of the whole property. They want to be able to look at the whole property and get a scale of the damage, then if they need to look in an area/sector with more detail, they can load the individual map corresponding to that sector. Therefore, the map of the whole property doesn't have to have nearly the same resolution, have geo information like a GeoTIFF. It just needs to show the orthomosaic for the whole property. It could even be downsized to a JPEG to make it more user-friendly. They'd also like to do a printed image of the whole property that they can put up on their wall.

In Pix4D you can run individual flights and then combine the individual flights into a single ortho. You are going to want to heavily compress the individual orthos before combining them. I tried it once and crash my workstation about midway through the process while trying to run full-size orthos. I have not attempted it since. Its easier to make a map service in ArcGIS and tile them...easier on the hardware requirements.

dronie326 · Oct 22, 2018

Following this with interest! Props for mapping that massive of an area. I can't imagine how long that must've taken.

Clinton Carman · Oct 22, 2018

Thanks Dronie, it took quite a while and was a bit more than we were initially expecting. Overall we did about 33 flights, each flight having about 600-1800 photos. The differences really depends on what the terrain allowed for with accessibility and line of sight issues. We were actually met with a lot of physical challenges with the weather. A large fire started about 50 miles away in the middle of one of our trips up there. That cut our trip short as the visibility dropped to less than 50-100 feet. Another time it started raining on us, and one other visit we had winds gusting at 25-30mph. Another big challenge we faced was having the proper light. As we were recording in the mountains, there were deep valleys and mountain faces that would be completely shadowed at different times in the day. With all that combined, it took us 11 days of flying to finish the maps. Now it seems we have a lot of processing time ahead of us as well...

Clinton Carman · Oct 22, 2018

R Martin said:
In Pix4D you can run individual flights and then combine the individual flights into a single ortho. You are going to want to heavily compress the individual orthos before combining them. I tried it once and crash my workstation about midway through the process while trying to run full-size orthos. I have not attempted it since. Its easier to make a map service in ArcGIS and tile them...easier on the hardware requirements.

R Martin, I said in my original post I had successfully completed the merging of two maps into one larger map, but like you said, it took me quite a while. I can't imagine trying to merge 32 maps into 16, down into 8, into 4, then 2, then 1. That would literally take months IF I had the computer power to even attempt such a thing.

I think what I'll do is process the items, get the ortho images in GeoTIFF format, export those to JPEG, then combine those in Photoshop/Lightroom into one single image. Then provide the GeoTIFF files of the individual sectors for close inspection needs. I'll also get the ortho tiles and import them into Google Earth with .kml files which should be pretty end-user friendly.

dronie326 · Oct 22, 2018

I was going to ask if you could just throw it to DroneDeploy or MME to do the combined processing but what you laid out above sounds like a pretty solid plan.

Clinton Carman · Oct 22, 2018

dronie326 said:
I was going to ask if you could just throw it to DroneDeploy or MME to do combined processing but what you laid out above sounds like a pretty solid plan.

Even at the highest paid levels, DroneDeploy only allows for 5,000 images per map really. Now I don't know about some customized job. And on MME I haven't seen a limit listed, but since they're require you to pay per map, that MIGHT get really expensive. I will look into it though.

dronie326 · Oct 22, 2018

Ah I knew they just released a new 'Large Maps' feature but I thought it was bigger. Looks like its 10,000 images.
You might be able to work out something with their Beta program though. Just a thought. I like your idea of stitching with photoshop though.

Product Release Wrap-Up August 2018 – DroneDeploy's Blog

Clinton Carman · Oct 22, 2018

dronie326 said:
Ah I knew they just realized a new 'Large Maps' feature but I thought it was bigger. Looks like its 10,000 images.
You might be able to work out something with their Beta program though. Just a thought. I like your idea of stitching with photoshop though.

Product Release Wrap-Up August 2018 – DroneDeploy's Blog

That's right! I had forgotten about that. IF I can get in on that, and it works, that may be the way to go. A lot easier to combine 4 maps into 1 versus 32 maps into 1!