The Revolutionary Government of Zanzibar (RGoZ) with the support of the World Bank has been developing the Open Data for Resilience Initiative (OpenDRI) with the aim of supporting evidence-based and innovative solutions to better plan, mitigate, and prepare for natural disasters.
Zanzibar is part of the Southwest Indian Ocean Risk Assessment and Financing Initiative (SWIO RAFI) which seeks to address high vulnerability of the Southwest Indian Ocean Island States to disaster losses from catastrophes such as cyclones, floods, earthquakes and tsunamis. These threats are exacerbated by the effects of climate change, a growing population and increased economic impacts.
The Zanzibar Mapping Initiative (ZMI) is a cooperative project between the Tanzania Commission for Science and Technology (COSTECH) and the Revolutionary Government of Zanzibar (RGoZ). COSTECH agreed to provide its in-house expertise and drones to support the creation of a new aerial base map of Zanzibar for the Zanzibar Commission for Lands (COLA).
UAV data has been collected over a large area +1,500 sq/km. The UAV’s used were small and as such the area was divided into 200+ zones each of which were then processed into individual Geotiffs. The data was collected without absolute accuracy and as so although the data within a given zone is relatively accurate there are varying degrees of edge matching issues when attempting to put all 200+ zones together.
A post processing process that would attempt to take individual zones and automatically do adaptive filtering of zones and then attempt to match using edge matching.
A 27 Zone section was selected as the same dataset.
3km x 3km Zone.
at a glance it looks good, so lets look a little closer…
We selected two zones…
|zone 117||zone 118|
Zoomed to 1:1000 Scale
Zoomed to 1:5000 Scale
|zone 117||zone 118|
Zoomed to 1:1000 Scale
|zone 117||zone 118|
1km x 1km Zone
Zoomed to 1:5000 Scale
Okay so I am finally writing this post. My initial target was to have it done by the end of January, but that slipped so next target was mid Feb, and that has slipped but better late than never!
Last year I together with Stephen Mather of OpenDroneMap were privileged to be invited to Kerala, India by International Centre for Free and Open Source Software (ICFOSS) to run a workshop on open source solutions for processing drone imagery and open hardware drone solutions. Checkout Stephen’s blog post here. As usual I was nervous, although I have been diving deep into the topics the past few years, I still get nervous when I am going to do a presentation or workshop.
Was a little hard leaving home, I don’t like leaving my girls for more than a couple of days, and even though this was only going to be 5 days, India felt like a long way away.
I left very early from home, but as usual the traffic in Dar es Salaam is totally unpredictably. By about half way it was clear I was not going to make it, so I started considering my options. Only one really, get a Bodaboda, or rather two Bodaboda’s, one for me and one for my luggage.
Got to the airport in time and was able to board the flight to first Dubai and then India.
Arrived in India quite late at night around 4am, as usual I was a little concerned about getting through customs with all the gear but it was no problem at all, even with the big ebee case(link), and the tons of drone bits in my bags. Once outside the airport the person who was there to collect me seemed to recognize me immediately, (maybe he was told to look for the black guy with dreads!). He took me straight to the guest house which was on the Trivandrum Technopark campus, and within 20 mins I was in my room and about to get some rest, (by then it was 5am), when there was a knock on the door! I opened and there was a guy standing, telling me that they are ready to leave in 30 mins…
Although the organization that had invited us was based in Trivandrum Technopark, India’s Largest IT Park, the actual workshop was to be held at Vagamon some 190 km a hill station at least 5 hours drive.I have to admit I was a little rough saying “NO I have to get some rest”, he went away for a while and then came back to let me know it was okay, I would go later with another group! Nice, I could get some sleep.
Woke up around 10am, and of-course top priority was INTERNET! Unfortunately the guest house did not have internet, which seemed kinda strange as it was on a science park, so went off wondering to see if I could get a sim card. After wondering around for about an hour and talking to a number of people it seemed that you need an Indian ID to get a sim card, not so sure that was true, but figured I ought to get back to the guest house in-case my host was looking for me. Lucky I did as someone from my host turned up a few minutes later and we headed together to the ICFOSS offices.
I spent a couple of hours at the ICFOSS office, got online, checked in with my family back home and checked my emails etc. Then headed back to the guest house because the plan was to drive up into the mountains at around 17:00.
The drive was great, a little long, but it was so nice to see some of the Kerala countryside, arrived pretty late partly because we had stopped along the way to eat, FYI what people in Kerala consider junk food, many would consider a healthy snack!
Once at the place we were spending the night Stephen and I spent some time planning the following day and then everyone got some much needed rest.
In the morning we headed to the place where we were going to hold the classroom part of the workshop. Before the rest of the attendees showed up we had some time to work on the Ranger, ahead of its maiden flight that was planned for the afternoon. We had good attendance, Arun M the lead from ICFOSS together with the core drone team, Deepthi, Vaisakh, and Asishwith, plus other ICFOSS staff, members of the Kerala electricity board, Veenus and Manoj from 1074 Vectors and others. All in all we had people from different sectors of government, startups and some more mature private sector companies.
Stephen started off the workshop with introductions and then it was over to me to talk about some of the work we have been doing in Tanzania. For the most part I covered how drones had helped us with the mapping of several flood prone areas of Dar es Salaam in the Ramani Huria Project, and how the biggest project yet the mapping of Zanzibar with low cost drones, had gone on to win awards and really helped validate all the work we were doing. It was great to have representatives not only from government, but also from the private sector, all keen and interested in how drones could help their particular needs. I left them with some thoughts about how to engage with the relevant government stakeholders to ensure that not only were they building technical expertise, but also ensuring a regulatory framework that would encourage innovation and be supportive of local businesses. The afternoon was supposed to be a field session were the group would actually see some drones in action. One of the guys from ICFOSS and I went hunting for a good place for take-off and landing.
We were lucky and found a great spot, so we started with out field setup, we actually had quite a few drones with us…
- Hobby King Duet. Small fixed wing, basically a park flier that can take a beating and keep flying making it perfect for training.
- SenseFly Ebee, a great platform even if its proprietary.
- DJI Mavic, easy to use consumer drone.
- Homemade quad-copter build around a pixhawk.
- 3DR Solo
- Ranger RX
The aim for the afternoon session was to demonstration how both a multi-rotor, (3DR SOLO), and a fixed wing (Ebee), can be used for mapping highlighting the difference and when each is most appropriate. Best case we were hoping to do some actual mapping with both platforms so that we could have some data that could be used for the OpenDroneMap processing demonstrations. Everything went perfectly, both the ebee and the 3dr solo were used to gather some data, and we even had some time to play around with the Mavic and really show the group how far consumer drones had come.
You may be asking yourself why we had a SenseFly Ebee at a workshop that was about OpenSource and OpenHardware? As it stands I still heavily recommend someone who is exploring the use of UAVs for mapping and other survey activities to get an Ebee IF their budget allows. I say this because the Ebee is a platform that just works, it comes with both the flight planning software, processing software, (Pix4D) and an excellent support infrastructure that is critical when you are getting into this field. Things will go wrong, and it’s nice in the beginning to minimize the things that could go wrong, and to have an organization that provides excellent support available, should you need it. There are many times when I have been out in the field doing a client survey and I have had to call SenseFly support to help get on some technicality or the other. Another reason why I recommend the ebee especially in countries where regulations are still in their infancy if they exist at all, is because of the extremely low risk the low weight of the ebee poses to people or property in the case of an accident, although I would not advise it there is even a video of a Danish firm doing a “test”.
That evening we all huddled together around the table planning for the next day. I processed the data we had collected during the day, Stephen and Deepthi planned the flights they expected to do the following day, and even decided to use the elevation data from my flights to plan other flights what we would do the following day. This was a great opportunity to show how valuable data generated from a quick drone survey could be, after just 2 hours in the field we already had elevation data that was more accurate than we could get from Google.
The following day we planned to do some quick flights in the morning and then head back to the city so that on the last day we could do some classroom data processing work. The flights went almost too perfectly, in my opinion you have not flown till you have crashed. The last flight of the 3DR Solo was done by Deepthi, and unfortunately, (or fortunately), there was a small issue during landing which broke 2 props, no huge issue and props are “consumables” anyway! This was also the last chance to test the , the area was not perfect, but we gave it a try anyway, Stephen on the controls, (the only one with real manual fixed wing experience), fact is though the runway was just not long enough and he had to abort at the last moment, one broken prop but the UAV still in perfect condition.
Friday was the last day of the workshop held at the ICFOSS offices. Usually such workshops start with lots of time being wasted installing software and actually getting the environment ready. Not at ICFOSS, they had everything ready, a server already setup with OpenDroneMap, and a room setup with laptops for the participants. I wish every workshop I was involved with went this well. I won’t go into detail but some highlights were:
1) Everyone was able to connect to the ODM server and create their account without issue.
2) Everyone after a little bit of instruction was able to upload their data-set, and get the processing done.
But that was not the most impressive thing, while Stephen was driving the workshop I was preparing a 3D model of the work that I had made from the data using Pix4D. I have to admit was I was quite looking forward to showing it off, until I turned around and saw something very similar on one of the participants screen. I wandered over to ask what it was and it turns out, it was the 3D mesh from the work he had just done with OpenDroneMap, being rendered on windows using meshlab. I was humbled, excited and impressed all at once.
We closed the workshop, with clear plans on how we would like to move forward, and exciting possibilities of future collaboration, and a very strong feeling that there is so much we can learn from each-other. That evening we had a great tour of Kerala with Arun and Jucar, which ended with a great meal on the beach. All in all I really enjoyed this trip, and looking forward to my next, (hopefully with my family this time), and to the great collaborations that I hope will come from it between Uhurulabs, OpenDroneMap and ICFOSS.
Sustainia just launched their 4th Global Opportunity Report together with DNV GL and the United Nations Global Compact. Over the years, the Global Opportunity Reports have proved that no challenge or risk is too big for business to tackle, and that there is market potential in every one of the 17 Sustainable Development Goals, from smart farming eliminating hunger to solar micro-grids providing clean energy.
All market opportunities accumulated in the Global Opportunity Reports are now available on our innovation hub Global Opportunity Explorer for everyone to explore. But which market has the biggest positive impact on people, planet and profit?
Get your copy of the report here.
So for a couple of days last week I went back to school. A few of us from africanDRONE, namely Unequal Scenes, Microdrone, African Defence Review and of course Uhurulabs, were sponsored by ICFJ to attend a three day intensive Drone Journalism School at the University of Oregon, in Portland.
The conference was organized by Google News Labs, the University of Nebraska School of Journalism and Poynter. It was a very interesting experience as for the most part my work with drones has been for data collection, survey etc, while most in the room were focusing on using drones for video and photography. One thing we all did have in common however was an overwhelming understanding that the rapid commercialization of consumer drones has changed the realm of what’s possible and by whom in ways we could not have thought possible five years ago.
A main focus of the conference was the training for the American FAA Regulations Part 107, which are the new rules for non-hobbyist small unmanned aircraft (UAS) operations in the USA. The conference had several great tutors who took everyone through all the aspects of the regulations in preparation for the attendees to take the exam.
Regulations are the topic of the day, and of great concern to anyone who wishes to operate UAS commercially. All over the world the various regulatory bodies are struggling to create rules and an environment that allow for the safe and fair usage of our skies. A colleague from africanDRONE has said that he thinks the American approach is “pragmatic and economically advantageous“, this of-course is relative to the South African regulations where it costs up-to 10,000$ to be fully registered and certified to operate UAS commercially. Contrast that with Tanzania where currently there is no payment required for the registration of UAS, there are only procedures you are required to follow with the Tanzanian Civil Aviation Authority (TCAA), and the Ministry of Defence (MoD). I personally have done this a number of times and have found the process straight forward. The one thing I will mention is that until now you have been required to do this for every flight, which can become a burden. I am currently in the process of getting a more permissive permit, that perhaps will only require me to log a flight plan whenever I want to fly, fingers crossed on that!
Lots of people know me as the Drone guy, what many don’t know is that I don’t think I am actually a very good drone pilot, I almost never fly for photography or video and much of my work is automated, where all flights are planned and programmed in the office. I rarely get my hands on the latest consumer drones and even when my friends offer me to try theirs I say no, as I don’t want to crash a $6,000 Inspire! Luckily for me one of the conference sponsors DJI, had brought a number of their latest drones and took us all out for some test flights. The inspire 2 is awesome, but would be a waste in my hands, happy though that I had a chance to finally fly it!
Although I have been using KVM, (Linux Based Kernel Virtualization), on my own infrastructure for some time, it is only recently, (past 4 years), that I have comfortable enough with it to use it in production for my clients, especially when other admins would be involved with administration of the systems. A choice I often made was to use Citrix XEN, as it…
- Has a commercially supported version should my client decide to stop using me.
- Has a fully free and open version which for the most part is not cobbled, at least it has all the core functionality that my clients need.
One thing that always frustrated me about it was the fact I needed to use a Windows machine to use access its management interface, yes there is a commmunity driven linux version but never worked quite well.
Anyway one of the clients where I had used Citrix XEN, is now ready for an upgrade and I found myself in a situation where I needed to move virtual machines from XEN to KVM. It took quite a bit of digging and experimenting but this is what I came up with.
First of all you need to gather three pieces of info:
[root@xenhost ~]# xe host-list uuid ( RO) : 181d1fa8-ef74-43ed-ac51-67c80717f6f0 name-label ( RW): uhu-xen-01 name-description ( RW): Uhurulabs XEN Server
uuid ( RO) : 52ce1f9d-faeb-f1e8-7054-505ac2ace647 name-label ( RW): Pool-wide network associated with eth1 name-description ( RW): bridge ( RO): xenbr1 uuid ( RO) : 2014eeb6-6f09-d3bf-88e4-1a0978ee8df7 name-label ( RW): Host internal management network name-description ( RW): Network on which guests will be assigned a private link-local IP address which can be used to talk XenAPI bridge ( RO): xenapi uuid ( RO) : eb57b4d8-7656-c607-b1cc-93cfd8766afe name-label ( RW): Pool-wide network associated with eth0 name-description ( RW): bridge ( RO): xenbr0
uuid ( RO) : fd6a4413-16ed-45ac-b206-c1587105ffb9 name-label ( RW): windows_srv_2008_std.iso name-description ( RW): sr-uuid ( RO): 05f8b9fd-438c-42ed-9da0-56c24c9ad13e virtual-size ( RO): 3166896128 sharable ( RO): false read-only ( RO): true uuid ( RO) : 5c9595df-f6d9-4a17-8de9-e53a611308f0 name-label ( RW): CentOS-6.0-x86_64-bin-DVD2.iso name-description ( RW): sr-uuid ( RO): 05f8b9fd-438c-42ed-9da0-56c24c9ad13e virtual-size ( RO): 1182699520 sharable ( RO): false read-only ( RO): true uuid ( RO) : 2713e717-8847-4d1c-abb8-4725a0ce1d88 name-label ( RW): THIS IS THE MACHINE WE WANT TO EXPORT name-description ( RW): Created by template provisioner sr-uuid ( RO): 4f02d7e3-67b6-3d14-9533-ebfb4fa323f8 virtual-size ( RO): 53687091200 sharable ( RO): false read-only ( RO): false
So now we know that
- HOST_UUID is 181d1fa8-ef74-43ed-ac51-67c80717f6f0
- VDI-UUID is 2713e717-8847-4d1c-abb8-4725a0ce1d88
- NETWORK-UUID is eb57b4d8-7656-c607-b1cc-93cfd8766afe
We now need to tell the XEN Server to put the machine in “transfer” mode, NOTE that this will make the machine unavailable for the duration of the export process.
xe host-call-plugin host-uuid=181d1fa8-ef74-43ed-ac51-67c80717f6f0 \ fn=expose args:vdi_uuid=2713e717-8847-4d1c-abb8-4725a0ce1d88 \ args:network_uuid=eb57b4d8-7656-c607-b1cc-93cfd8766afe \ args:transfer_mode=http plugin=transfer
As we are going to do the transfer over HTTP we need to find out the correct URL to access
xe host-call-plugin host-uuid=181d1fa8-ef74-43ed-ac51-67c80717f6f0 \ plugin=transfer fn=get_record args:record_handle=af2eed01-b162-b37a-982c-4536dd2b5bc3 ?xml version="1.0"?> <transfer_record username="47b5ebf70160c752" status="exposed" url_path="/vdi_uuid_2713e717-8847-4d1c-abb8-4725a0ce1d88" record_handle="af2eed01-b162-b37a-982c-4536dd2b5bc3" device_2713e717-8847-4d1c-abb8-4725a0ce1d88="xvdb" url_full_2713e717-8847-4d1c-abb8-4725a0ce1d88="http://47b5ebf70160c752:email@example.com:80/vdi_uuid_2713e717-8847-4d1c-abb8-4725a0ce1d88" ip="192.168.13.88" transfer_mode="http" url_path_2713e717-8847-4d1c-abb8-4725a0ce1d88="/vdi_uuid_2713e717-8847-4d1c-abb8-4725a0ce1d88" all_devices="xvdb" url_full="http://47b5ebf70160c752:firstname.lastname@example.org:80/vdi_uuid_2713e717-8847-4d1c-abb8-4725a0ce1d88" device="xvdb" use_ssl="false" password="f2b25a5854c12b76" port="80" vdi_uuid="2713e717-8847-4d1c-abb8-4725a0ce1d88"> </transfer_record>
Now we have the URL: http://47b5ebf70160c752:email@example.com:80/vdi_uuid_2713e717-8847-4d1c-abb8-4725a0ce1d88, we can now use curl to get the image
curl http://47b5ebf70160c752:firstname.lastname@example.org:80/vdi_uuid_2713e717-8847-4d1c-abb8-4725a0ce1d88 -o machine_raw_disk.raw
You can now import machine_raw_disk.raw as a new image into KVM!
I have been noticing with more and more concern the increasing amount of what looks to me as VERY young police officers on the streets carry what look to me like AF-47, (I don’t know much about guns!!!). When I have seen what feel to me like “gangs” of them on street corners I have honestly asked myself if I feel more or less comfortable. Fact is they make me nervous as hell, and part of me has been comforted by the fact that I figured that probably most of the guns are none functional or not loaded. I came across a video yesterday that confirmed that I should be very afraid, AND that my hope that the guns are not loaded or not working was crushed.. indeed they are loaded, they do work, and unfortunately looks like the police are somewhat lacking in their training.
A few months ago my home firewall, (pfsense) died, was built in an old Dell Optiplex 755 that I picked up second hand for about 300,000TZS (150USD), it had served me perfectly for over 2 years so was not too upset when it died. I threw in a Zyxel UG20 firewall device that I have had hanging around, and I have not been happy, but more on that in another blog post. So today I decided to go through some of the junk I have around to see if I could not build another pfsense firewall, anyone who knows me knows I have lots of junk. I settled on an old HP chassis I have, major overkill as its 8GB and even found a SSD in it. This used to be my primary workstation, (after our office was broken into about 10 years ago and my very very very special machine was stolem 🙁 ), it then became my home server, and then about 2 years ago was retired, (figured I at the time I would repurpose it, but did not think it would take this long).
So yes its major overkill for a home firewall, but I will put a whole bunch of other networking tools on it….
So firewall is installed and in place, now for some quick iperf tests but figure it might be worth documenting, (as I always have to do a refresher, on what to expect)…
|Network Core||Which is..||And means in theory the fastest you can transfer data is…|
|10Mbps||10 megabits per second||1.25 MB/s|
|100Mbps||100 megabits per second||12.5 MB/s|
|1000Mbps (1Gbps)||1000 megabits per second||125 MB/s|
The iperf test was run between the new pfsense server and one of the pfesense boxes that is only 5 hops away:
------------------------------------------------------------ Client connecting to X.X.X.X, TCP port 5131 TCP window size: 65.0 KByte (default) ------------------------------------------------------------ [ 3] local Y.Y.Y.Y port 7131 connected with X.X.X.X port 5131 ID] Interval Transfer Bandwidth [ 3] 0.0- 5.0 sec 6.00 MBytes 10.1 Mbits/sec [ 3] 5.0-10.0 sec 6.00 MBytes 10.1 Mbits/sec [ 3] 0.0-10.0 sec 12.0 MBytes 10.0 Mbits/sec
A second test using www.speedtest.com, also show a max transfer of 8mb/s up and 10.1 down, so its safe to assume we are being capped at 10mb/s. Now though I would like to find out if that is a ISP cap, or there is a 10Mb physical link somewhere…
The idea was to merge into 1 geotiff 28 irregular shaped geotiffs which ranged in size from 1g up to 13g and making up a total of 113g. Why? Because I then needed to cut the resulting geotiff into multiple irregular shaped individual geotiff’s. This took a couple of days, and I was quite keen to come home today as this morning it was at 80%, you can only imagine my disappointment when I checked and found…
0...10...20...30...40...50...60...70...80...90..Traceback (most recent call last): File "/usr/bin/gdal_merge.py", line 540, in <module> sys.exit(main()) File "/usr/bin/gdal_merge.py", line 526, in main fi.copy_into( t_fh, band, band, nodata ) File "/usr/bin/gdal_merge.py", line 270, in copy_into nodata_arg ) File "/usr/bin/gdal_merge.py", line 63, in raster_copy nodata ) File "/usr/bin/gdal_merge.py", line 105, in raster_copy_with_nodata nodata_test = Numeric.equal(data_src,nodata) MemoryError
This as you can imagine was very frustrating, and I just assumed that the output was junk but figured what the hell, its created a 301G file lets see what it is. Started the process to load into Qgis, and since I figured it would take a while started this blog post. It seems though that output might be useful as qgis eventually loaded the file, and it actually looks like what I expected. So first things first I have set qgis to now save the file under a new name… Its going to take a while, currently at 12.2G and I actually expect it to end up bigger than the initial 301G
… a day or so later …
So it turns out the merge worked, I will revisit why the error later, but the file saved by qgis had the exact same size, and has the same result using gdalinfo. AND i have just done my first clip BUT it seems I made a mistake the command I used was
gdalwarp -dstnodata 0 \ -q -cutline shape_file_to_clip_to.shp -tr 0.05806 0.05806 \ -of GTiff input_file.tif \ clipped_file.tif
Which resulted in a file the same size of the input file, and the same dimentions, what I should have done, I think, is:
gdalwarp -dstnodata 0 \ -q -cutline shape_file_to_clip_to.shp -tr 0.05806 0.05806 \ -of GTiff input_file.tif \ clipped_file.tif
Which is what I am running now, the previous command took about 3 hours!