New Firewall Status!

So a couple weeks ago I wrote about the new firewall setup.

Quick update on the status of that.  I built it out.  Got it working.  yay!

294989But I want more cool stuff.  So I have a business class fiber line run to the house with 5 static IP’s.  So I decided to see if I could get pfsense to do failover router.  You know, because why not.  I tried to get pfsense to install on this old thin client I had.  No go.  WAY TOO OLD!!!

So I moved on.  I too my esxi box and with the second nic ran it to another small unused switch of mine.  I then configured esxi with a second virtual switch that uses that NIC on the new physical switch.  I know, I could do that with vlans.  Duh!  But I am waiting for a cable for my switch so I can configure said vlans.

vSphere ClientSo with this new physical port I was able to build a pfsense box in my virtual environment.  I configured it with dual nics, 1 in each switch.  As you can see here.   My routers are named voot1 and voot2.  Named after the voot runner from the show Invader Zim.   My main network scheme is based on characters and vehicles from this show.

So now I have a physical router configured with one of my public ip’s   x.x.x.253.  Internal its configure as x.x.x.2.  I then configured my CARP ips.  You create a virtual IP for both lan and wan.   External is x.x.x.250, internal is x.x.x.1.

pfsense - New Page (1)

Set the outbound nat to use the .250 address and we are golden.  Setup all the carp sync settings to push the settings to the backup pfsense.  This works BEAUTIFULLY.   I was so amazed as how easy it is.  I just make a change pretty much anywhere in the main router and its automatically sync’d to the backup.

Once the backup is sync’d it’s time to test.   Reboot the primary.  Network dropped and didn’t come back up until the primary fully booted…

stewiegriffin-familyguy

Ok.  I got it.  The arp change caused some problems with the esxi side.  There is a switch security feature that blocked it.  You have to set the switch to accept promiscuous mode….. $%^& I think my switch is a slut.samantha_jones_promiscuitypromiscuous

 

Ok.  So now that my switch is acknowledged as a slut, it works.  Reboot primary router and the backup comes online, takes over the gateway x.x.x.1 and the external x.x.x.250.  This is awesome for various servers that I have in different datacenters that have ip security configured.

So this all went great, but the original reason for the upgrade was my old hardware choked itself out with my ipsec tunnel to AJ doing the encryption/decryption.  So we need to get the tunnel back up.   We got some time over this weekend to get the tunnel working.  We now copy back and forth at 30mbps.  30!!!!!   Fully maxing my fiber line over a secured tunnel and my cpu was only at 25%.  WIN!!!

 

BUT!!!!!! We have this awesome failover working…. will the tunnel come up on the backup too?   *checks the backup router* well the ipsec tunnel config was sync’d.  So what happens if I reboot the primary now….

NO EFFING WAY!!!!!  My tunnel re-establishes to AJ on the backup router too!!! HAHAHAHA this is awesome.

So Aj had asked me about a raspberry pi project on friday called pi-hole.  From the website..

“The Pi-hole is an advertising-aware DNS server that prevents ads from being downloaded. Once installed, configure your router to have DHCP clients use the Pi as their DNS server and then any device that connects to your network will have ads blocked without any further configuration. Alternatively, you can manually set each device to use the Raspberry Pi as its DNS server.”

But it seems like something that screams virtual machine.  I didn’t want to tie up a raspberry pi just for dns.  Turns out pi-hole doesn’t require pi hardware.  So it is possible to put in a vm.  So I configure a tiny ubuntu server and install pi-hole.

curl -L install.pi-hole.net | bash

Very simple.  But it works.  So I build out the vm.  Export it as an ovf and ship it off to AJ over the new tunnel.  He brings it up in his virtual environment and gets it working on his end.  Now to update our respective DHCP servers to tell our networks to use the new adblocking dns.   But… what if it’s down.  Ads/banners coub-1137363

There is a tunnel…. use each others as a backup!!!  Beautiful.

No my pi-hole is the primary on my network, AJ’s is secondary, and if both are down for some reason the network defaults to google.  services_ DHCP server

How well does pi-hole work?  Why do I keep saying pi-hole?  It sounds dirty.

pi-hole pi-hole pi-hole pi-hole pi-hole pi-hole pi-hole pi-hole

Yep.  Still sounds dirty.  But I got distracted.

Pi-hole Admin Console

That is insane…. 1726 ad’s blocked today.  We were gone for 4 hours and it’s still that high.  Crazy.  But you should see some sites without their ad’s.   Interesting.

This week/weekend has been pretty cool for getting some network stuff done.

 

New Firewall Build

So I have been running my home network on a home built router for well over a decade now.    Many moons ago when I lived in my apartment a client of mine was working on converting Eckerds to CVS stores.   As part of said conversion he had a ton of this little MaxTerm thin client machines.

3759951495_91064ca6e9

Which was not bad at all.  It has a little bit of memory, enough cpu to run as a router and an expansion slot.   Throw an extra network card in there and there you go.  Next to no power router.  I have been running http://m0n0.ch/wall/index.php since the beginning.  M0n0wall is now a dead project.  It will be missed, it sparked some of the great pieces I use today.   pfsense, freenas ect.   I have the last version on my little thin client and it runs well.

geek-is-coolBut like everything, eventually you run into something that makes you move on.  I had this recently.  Remember the great data apocalypse of 2015?  Well, that has sparked many changes in my backup replication plans.  One being that a buddy of mine and I are building secondary NAS systems for our freenas to put in each others server racks.  Yes, my buddy is a geek too.  So as part of this secondary nas setup.  We are going to be doing replication across the WAN.  Even though it uses SSH communication to do the replication.  We both felt doing an encrypted tunnel between locations was better.  Plus would let use gain access to each others resources without having to open unnecessary ports.  So we setup a nice encrypted tunnel.  Time to test some transfers…..

150kbs a second……..What-Meme-13

Now I never said I was a smart man.  But there is something wrong there.   AJ’s wan is 150/150 MEG!!!! (thank you FIOS) and mine is 50/50 also on FIOS!!!

No no no.  No sir, I do not want another.  I want the one I originally asked for!!!!!

Who broke that chair!!?
images

Ok.  I am calm now.  WTH is going on.   Checks the traffic graphs…

That’s not it.

Checks CPU graph on thin client….
hell-fire-1

Stops file transfer and checks again….
i_call_upon_the_power_of_ice_and_snow_by_fromzerotohero-d60aqs9

What the hell!?!  Now who broke THAT chair!!!!??

4565396395_c89b9c22f3_b

So it would seem that doing the encryption is causing a problem.  The cpu is just choking on the encrypt/decrypt over the tunnel.  That kind of sucks.  However we are also running on an old piece of hardware (12 years old?) and running on and EOL firmware.  Ok.  New Router time.

So this weekend I begin working on my new pfsense router.

Using this Intel D2500CCE Atom D2500 Dual LAN motherboard.

51mMt8cv6CL

Amazon

With a 2 gig memory chip this is ready to go.

I will post an update after I have this in and running for awhile and testing the encrypted transfers.  To see if it really is all in the cpu.

Some projects and upgrades

So I have a few things on order coming from china for some various projects.

I figured since I haven’t posted much lately I will post some pre-project dribble.

Phoenix Connectors – Aliexpress

phoenix connectors I have these coming to be able to connect audio up to my ClearOne XAP800.   ClearOne XAP-800 Professional Audio Conferencing System Pic 4 xap800_l

The XAP800 is going to be added to my Home Automation system to be able to control/mix/route audio from various sources to various destinations.  If you want to read more about what this can do check out my buddy here.

http://hazymat.co.uk/2015/04/multi-room-audio-options/

 

3d Printer Upgrades

Its funny.   The whole time I have had my printer I have been constantly wanting to upgrade.  Maybe because I built my printer from a kit and know each piece.  I look at upgrades thinking, this will help just a tiny bit more.  This will make it a tiny bit better.  But also, I enjoy these projects.  So I have what I can only hope are my last big round of updates.  Nevermind.  I just thought of one more.  Bowden extruder…. but that’s for another post.

Motor Coupler – Aliexpress
The motor coupler is going be to used to connect my 5mm drive shaft to my below 8mm threaded rods.

Lead Screws w/ Copper Nut – Aliexpress
Replacing my current threaded rods and nuts with some nice thick Lead screws and longer copper nuts should add some additional stability and durability in that I won’t have to replace the nuts regularly like I do now.

Aluminum Build Plate – Aliexpress
The aluminum build plate will replace my glass bed.   I have hard it does well for heat distribution but also I need it for the below proximity switch.

Induction Proximity Switch – Amazon
I am going to get rid of current auto level setup.   I am going to the inductive sensor so there is no longer going to be a swinging arm.  This will hopefully improve start up time, and keep better accuracy.

shoppingUT82quJXalbXXagOFbXSReprap-Prusa-i3-3D-printer-parts-Anodized-Aluminum-BUILD-PLATE-for-Heated-Bed-3D-Printer-RepRap.jpg_640x64061fxf9nPCFL._SL1100_

 

Random

20 555 Timers – Aliexpress
ne555I have these 555 timers coming because I plan to use them on a small solar charge controller project I need for my Green Sprinkler system Project.

 

100 Optocouplers – Aliexpress
111781498246_1A couple weeks back I designed and tested a small circuit that will detect when the 12v accessory line in my car is turned on and off.  Using these optocouplers I will be able to use this signal to either power on my raspberry pi in my car, or to tell it to turn off.  This is going to be useful so that the pi doesn’t turn off as soon as I get home, but will be signaled to turn off in about an hour.  This will give enough time for sync jobs to run, backups and any config changes I want to do.

So there it is.  Some of my up coming projects

  • Big Printer upgrade
  • Solar Charge Controller
  • CarPi Power circuit deally (i need a better name)
  • Green Sprinkler System
  • Home Automation Audio Router

juggling-businessman-image

This guys has his shit way more together than I do.  Look at him!  He’s wearing a suit….  I barely wear pants!!!!

 

 

 

 

 

 

MineCraft Nightlight Christmas Present

Ok, so my son is 8 and that means he automatically likes Pokemon and minecraft.
He also likes anything with LED’s (geek in training I believe… blinky oooooh shiny…. sorry, distracted.)

So awhile back I came across this post BLE controlled Minecraft nightlight.  I figured, this is perfect.  So I started order some parts (making a few changes of my own along the way) and printed out the cube and base.

Parts:
Adafruit Pro Trinket – 3V 12MHz
Adafruit Neo Pixel Strip
Capacitor
3d Printer
Adafruit BlueFruit LE Bluetooth module

 

2015-12-16 17.56.55

 

2015-12-16 17.57.19 2015-12-16 17.57.23

I then soldered up the led strips I had from a different project.  Finished wiring everything up based on Mr. McMillan’s guide.  One of the changes I made was I needed to make a different base piece.   The cover originally designed required some larger feet than I had for it.  So I designed a new piece.

Here is the alternate piece.  Thingiverse

2015-12-16 21.00.042015-12-16 21.00.012015-12-16 20.59.592015-12-16 20.23.34

Had an issue the weekend before christmas and had to order some replacements.  So here I was Christmas Eve soldering everything up.

2015-12-24 16.57.26

And I got it all working just in time.  So Christmas morning it was sitting beside all the presents already lit up.  I didn’t get a pic of it under the tree.  Kind of wish I had.  But here is a pic of it light up on my desk.

2015-12-24 17.50.24


Storage Problems (UPDATED!!!)

So last week my big storage box started acting up.  Random reset, dropping a drive, all and all, not good.

So let me give you a quick rundown of this storage box.  I am running freenas.  I have a total of 11 drives currently.  9 of these drives are 2TB drives.   Configured in 3 Raid 5 configurations.   There is a small OS drive and a 128gig SSD just for cache.  Then striped across giving me a total of 9.63TB of storage with redundancy.   I store everything here, all my video and photo work.  My media collection.  My ESXi environment mounts iscsi off this thing.  So it’s pretty critical my geek life.

I did all sorts of testing.  Flashed the OS drive.  Replaced the OS drive.  No matter what I did.  4 minutes uptime, kernel panic and reboot.

So I ordered new parts which arrived yesterday.  I take the system out of the rack, put it on the table, open it up…. found the problem…

IMG_0398

Ouch.  A small fire in my server.

 

Update!!!! (12/23/15)
So things went from bad to worse.  Shortly after finding and fixing this.  I reinstalled the OS and brought everything up for a 24 hour burn in.  This worked.  Ok good, lets go back to the SD card for the OS.  Fresh install, 24 hour burn in.  Lets go!!

12 hours in.  System reboots.  Doesn’t come online… No prob, Ill fix it when I get home…….. (do you see the foreshadowing here?  cause I didn’t)

I get home, not booting right…  Ok,  reinstall os…. nope.   ok, maybe sd card is bad.  Back to the SSD.   Nope..

Uhhhh WTF!?!?!

Clean OS.  No auto Import.  Everything is fine…. import zfs volume…. kernel panic.  Dead..

1234931504682

Time to research.   Ok so from the inter-webs my prognoses is “screwed, data gone.

original

Apparently desktop memory and zfs are to blame here.   Not like I wasn’t trying to keep my data.  I had 3 raidz vdevs in a zfs pool.

So after contemplating all my poor poor data I decided to try to recover it.

Disk scans (SpinRite for 36 hours)  = nothing
zdb scan (multiple hours but kept crashing because ran out of swap) = nothing
OpenIndiana live cd = nothing

Finally I found a post where someone talked about trying to force the volume only as read only.  I figured, “hey, I’ve already spent 4 days trying to recover, why not”

So I boot up freenas.  Get on the console and type


zpool import -f -o readonly=on -R /mnt vol

It didn’t kernel panic….. wait, what?!

Holy $%^&*    IT MOUNTED!!! I’m jumping through directories all giddy that my data may still be intact.   But read only isn’t going to do me much good.  Need drives!!!!

I don’t have 10tb of external drive…. AJ!!!!

So I go to my buddies and steal all his externals.  I plug the all in at once and start the very very very slow copy.   After 5 days of copying to externals I was finally able to rebuild and start putting my data back.

So now the lessons learned:

  1. Regularly check that your offsite back ups are working
  2. Build a secondary nas for snapshot backups (this box will eventually be at AJ’s since we have a VPN between our places)
  3. Identify what is replaceable and what isn’t and dump that somewhere else too.

This was a long process but its coming to a close.  I will be doing snapshots of critical data to a secondary freenas box.   Once the initial snapshot is done, I will take the box to AJ’s and the snapshots will continue to backup there.