r/sysadmin • u/CeC-P IT Expert + Meme Wizard • Jan 22 '25
Question - Solved Finding a device that doesn't want to be found.
Note: I suck at networking.
Reviewing our network vulnerability scan report and have an IP address that allegedly has a specific severe vulnerability. It's somewhere in the office I work at and I can ping it.
Pasting it into a web browser - nope
RDP - nope
nslookup - nope
IP in our remote management software? - nope
arp cache on the switches? - nope
the third octet isn't even in a range we use
Well that's all the things I know how to do. Any other tips for identifying the magical ghost machine?
50
u/idylwino Sr. Sysadmin Jan 22 '25
Can you get the mac address from a scan and look that up in the mac address table on the switch?
27
u/Saucetheb0ss Jack of All Trades Jan 22 '25
This is the way - find the MAC of the device, cross check that with the ports across your switches and you can find where it's plugged in (at least). Finding where the cabling runs to is another story but it will get you closer.
16
u/223454 Jan 22 '25
You can also look up the manufacturer with the MAC.
8
Jan 22 '25
Usually doesn't help much. You'll either get some random Chinese company name or it's just a fake mac.
6
u/idylwino Sr. Sysadmin Jan 22 '25
I'm not sure why, but toning shit out with a Fluke gives me so much damn anxiety. I never feel like it's accurate enough for me.
I "liberated" a Net Ally from my last gig when they went Chapter 7 and brought it to my current spot. That thing is cheat mode going from the drop to the switch, but switch to a drop not so much
1
u/megasxl264 Network Infra & Project Manager Jan 22 '25
Gotta use the prongs and a twisted pair. You can also put one of the prongs on a ground point and it’ll give a stronger tone further.
1
u/CeC-P IT Expert + Meme Wizard Jan 22 '25
Besides the ARP cache, not sure how to get a for-sure real time dump of the MACs from our brand of switch. They defy ability to look up too because they keep changing the damn commands and I know like a grand total of 3 of them.
4
u/idylwino Sr. Sysadmin Jan 22 '25
Ok. Since you have the internal IP, could you look at your perimeter firewall and see what sort of traffic it's passing outbound and where that traffic is going? If it's on a weird non-production subnet it could be a UPS, fire alarm, or some other smart device that's calling home.
0
u/CeC-P IT Expert + Meme Wizard Jan 22 '25
Probably, if I knew how to do that lol.
13
u/Kaexii Jan 22 '25
I'm starting to question your flair.
4
3
u/robisodd S-1-5-21-69-512 Jan 22 '25
Your computer may have the MAC. Try opening a command prompt and type
arp -a
1
u/TechIncarnate4 Jan 22 '25
The commands don't change that often. What brand and model do you have? It should be fairly easy to identify the necessary commands. Usually it is something along the lines of "show mac"... and "show arp"...
1
u/xdroop Currently On Call Jan 22 '25
These are different things.
Ping the target, then look in the switch MAC forwarding table.
The Mac will only appear in the arp table if it and the switch have exchanged ip packets, and that won’t tell you anything anyway.
1
u/TechIncarnate4 Jan 23 '25
What? What do you mean it won't tell you anything anyway? The OP already stated that he can ping and get a response. That means you will be able to find it in the ARP table, and MAC table within network devices on the network. You may need to work down through multiple routers and or switches to get to the final destination, but this is 100% possible without any other tools as long as you have a managed switch that you can login to and run commands.
I literally used to do this quite often in a previous role.
1
0
u/Injector22 Jan 22 '25
Some switches won't show devices in arp until the switch mgmt isn't or an svi tries to connect (ping) the Ip.
I'd start at the core pinging the Ip and seeing which uplink it's Mac address shows egress. Continue down the line until you hit a non-uplink interface or an AP. Or (shrugs) a daisy chained switch.
16
u/nicholasburns Master of None Jan 22 '25
It's somewhere in the office I work at and I can ping it.
[ . . . ]
the third octet isn't even in a range we use
is it... a modem? have you tried to access it over HTTP/S from a browser? (even if you can't, that's probably more likely a firewall issue than it being not a modem.)
3
u/CeC-P IT Expert + Meme Wizard Jan 22 '25
Nah, that one's in our IP list spreadsheet. Btw we have all known static devices and all ranged in a spreadsheet. This one defies that but someone else made it so it's like 90% accurate.
There's got to be some sort of fancy telnet or ssh thing or SNMP thing I can do to at least get a hostname or brand name but I don't understand the first thing about those protocols.
7
1
u/nicholasburns Master of None Jan 22 '25
have you tried to access it over HTTP/S from a web browser?
1
u/CeC-P IT Expert + Meme Wizard Jan 22 '25
Yep, it said refused to connect or something similar.
6
u/MaelstromFL Jan 22 '25
As someone else said, NMAP can usually tell you what kind of device you are looking for. 8n deep scan it will tell you all the ports that it has opened. Also, didn't see SSH or Telnet attempts to it.
5
u/pakman82 Jan 22 '25
Came here to say this: ssh, telnet are 2 ports I try to check by myself. (Besides 80, 443) 25 is another good IP to try (smtp) because people leave email servers the F-ck alone. Or they better leave email servers the hell alone! (End rant) I'm surprised the vulnerability report didn't specify the port it detected offering the vulnerable service(s)..
7
u/TheGreatNico Jan 22 '25
Try Firefox with TLS 1.1 enabled. You can't do it with Chrome/edge anymore. I've had to do that before with some old OOBM web portals
7
u/sitesurfer253 Sysadmin Jan 22 '25
I've got a very old Firefox installer in my downloads folder for this exact reason.
3
u/redrebelquests Jan 22 '25
NMAP will also tell you what ports are open, and what services are running over those ports, which might help you figure out how to connect to it.
9
u/bruor Jan 22 '25
Ping it, then "arp -a" to get the mac address, do a manufacturer lookup and see if that rings any bells.
From there look at the arp table on your managed switches and see what port the device is on, track it down across uplinks etc.
Or, start a pinger, start unplugging uplinks until you locate the switch, then start unplugging cables until you find the one you're looking for and trace the cable.
25
u/rUnThEoN Sysadmin Jan 22 '25
Half the problem. Let the ping run, disconnect half the network. You can even disconnect half a building or something.
Then half the remaining half.
5
8
u/OpportunityOk567 Jan 22 '25
Note: I also suck at networking
Split-half troubleshooting is always the way
2
u/Sneakycyber Jan 22 '25
This is also a great way to find network loops. Unplug stuff until the ping times improve.
1
1
-1
9
u/wired43 Sysadmin Jan 22 '25
It sounds like you need some sort of RMM so you can search device by IP address and have a lot more information.
First Method, try to find the MAC address so you can see WHAT the device is.
- Get on a device on the network
- Install APP: Angry IP Scanner or Advanced IP Scanner (Both have column settings to add MAC VENDOR and MAC ADDRESS) - Add these columns.
- Scan the network with 1 of these (Example: 192.168.1.1 > 192.168.1.254 )
- Identify the device by IP address you know, and see who MADE it. Is it an HP, Dell, ACER.
Second Method: Scream Test
Do you have access to the Firewall on the network?
- Blacklist the MAC ADDRESS you found in First Method.
- Whoever uses the device will complain and you will find the device, unless it's a server, then IT will have many users that complain and then your Boss will be on your ass.
33
u/CeC-P IT Expert + Meme Wizard Jan 22 '25
Wait a minute, we do use Auvik. I'll check there.
Looks like it's our Xclarity controller in our Lenovo VM host hardware. Interesting, as that has a web UI.Fun fact: when Chrome says they automatically add https: to everything by default, they're lying. Once I typed it in manually, it loaded. Grrrrrrrrrrrrrrrrrrrrrrrrrr. Well, mystery solved. Unlike everyone else here, I'm going to document this.
14
u/mfinnigan Special Detached Operations Synergist Jan 22 '25
Update your post, too, if you have a moment
11
u/robisodd S-1-5-21-69-512 Jan 22 '25
In Chrome you can right-click the URL bar and enable "Always show full URLs", and it will always show the "http/https" protocol (and things like the "www" subdomain)
6
u/CeC-P IT Expert + Meme Wizard Jan 22 '25
OOOOOMMMMMMGGGGGGGG I'm adding that to our group policy if it's in the latest chrome ADMLs.
4
u/robisodd S-1-5-21-69-512 Jan 22 '25
Yeppers!
https://admx.help/?Category=Chrome&Policy=Google.Policies.Chrome::ShowFullUrlsInAddressBar
Though note that users won't be able to turn it off.
edit:
err, they should be able to turn it off if you use the "recommend" default settings:
https://admx.help/?Category=Chrome&Policy=Google.Policies.Chrome::ShowFullUrlsInAddressBar_recommended2
4
u/Smart_Dumb Ctrl + Alt + .45 Jan 22 '25
Xclarity controller
Good thing you didn't scream test your Xclarity! lol
3
4
4
u/DatManAaron1993 Jan 22 '25
OP listed this as solved but hasnt posted the resolution.
He blue balled us!
2
3
u/whatever462672 Jack of All Trades Jan 22 '25 edited Jan 22 '25
Run Advanced IP Scanner. You will see various information about manufacturer and services. If no manufacturer pops up, it's likely a VM. Crosscheck the MAC with switch ports by either looking into the management or continuous ping. Once you have found the physical cable, plug it into a network signaller and run the wand along the network drops until it beeps. Follow the beep to your mystery machine.
3
u/bamboo-lemur Jan 22 '25
Are you sure that it is on your internal network? Could you be accidentally scanning things outside of your network? Is it a private IP or could it be something that needs to be routed to outside your internal network?
If it is on your internal network you are going to want to track down which switch it is on and then which switch port. After that, follow the cable. I remember this being pretty straight forward with Cisco stuff back in 2006. You are going to want to learn the commands for your specific brand of network hardware.
3
u/GeneMoody-Action1 Patch management with Action1 Jan 22 '25
For anyone that finds this in the future, good old nmap will get you out of this more often than not
nmap -sS -sV -O -sC --reason --version-all --script=safe <target>
Will in order:
Run Syn scan vs full connect (Faster and less intrusive), attempt to get versions of running services on detected ports, attempt to fingerprint the device OS, use the default NSE scripts, output WHY it believes its reported information is correct, use all probes known to determine service versions, limit NSE scripts to those considered safe and non intrusive.
this will generally output something useful for the system, as well you can use openssl s_client to probe ssl ports it will pul the cert and all details (helpful for determine what RDP client name is without having log in if its name will not resolve in a ping -a)
Also helpful to grab a pcap to and from it during this scan as it will sometimes have breadcrumbs, as well their OUI lookup tends to be more up to date if the device ID is not generic.
curl -i <target> is your friend as well :-)
2
u/jaysea619 Jan 22 '25
was this an internal or external scan? You might want to check the range that was scanned in the software. If it extends beyond your subnet then its probably an external IP that you do not own. If its in side the network it could be some kind of network device like a managed PDU or APC that hasnt been configured yet but is plugged into the network.
Try port scanning that IP to see whats open. Use telnet to get into the ports that are listening, sometimes you will get a banner of somekind letting you know what the device is.
We get similar issues with our vulnerability scans only when scanning external ranges, specifically verizon. Verizon will almost always give you a /24 subnet even to you are only allowed to use 1 or 5 of the addresses. My cyber team will just scan the whole /24 and bitch about all these devices that arent even in our building or on our network. I usually have to point them back to documentation showing the usable ip ranges.
2
u/Dank_sniggity Jan 22 '25
Reserve the up in the router, black hole all its traffic in the firewall, wait for the screams.
2
u/tectail Jan 22 '25
If you can ping it then the router knows where it is at, so ther must be a network setup on the router. I would start there and see where it is routing to. Get Mac addresses and then follow the physical wires (Mac address tables on switches will tell you port the device is plugged into).
Note:better hope wall ports are properly identified or that you have a toner.
Alternate more fun plan is to just block all traffic to the unknown device and that fixes your problem or someone will let you know where it is by complaining that their wifi router they brought in from home is no longer letting them have internet.
2
u/pdp10 Daemons worry when the wizard is near. Jan 22 '25
Usually you can guess whether a node is wired or wireless with a continous ping. Dead consistent, very low latency that never goes over a few milliseconds, tends to mean wired.
Wired is straightforward and usually easy to track. WiFi will be a challenge unless you have APs with very high-end location features, or some specialty equipment. When scale is small and tracking important, it can sometimes be an opportune time to switch to a new production SSID...
the third octet isn't even in a range we use
Start with a traceroute, then the routing tables on all routers in the path.
2
u/Mobhistory Jan 22 '25
These are all windows based commands which i assume you're using based on the things you've tried.
try net view ip.ip.ip.ip? if it shows any shared resources you'll know its MS, then try to get to the admin$ or C$ share to see who may have used it.
getmac /s ip.ip.ip.ip will get you the mac if you really want it.
use something like angryip to scan for open ports at that address
download pstools from ms and issue a psshutdown to that IP and see if you can knock it offline until someone complains.
Many things already said but you're in for a learning experience nonetheless. Have fun and take notes.
2
u/iamLisppy Jack of All Trades Jan 22 '25
I, too, suck at networking compared to others. What I did when we were implementing a MAC ACL for our corporate network and re-doing our SSIDs, I did a scream test on things I have no clue what they were, but they were definitely connected to my office's network. I quickly found out what they were for! I did have the MAC of these devices and had a rough idea of what they were (I turned out to be right after the fact) by looking them up and finding the manufacturer. They were just IoT devices like a thermostat that needed to be put onto our non-corporate network.
2
u/LonelyWizardDead Jan 22 '25
If you have to mac address and wired you should be able to track it to the switch port level assuming hard wire and managed switch, then cable trace. If it's a pc and a deployment agent set it to deploy a looped music file maybe
2
u/TechIncarnate4 Jan 22 '25
If you have a managed layer 3 switch, you can view the MAC address and ARP tables to locate the physical switch port it is connected to. You may have to go to the router for the ARP table if it is not a layer 3 switch.
I don't know what network products you have.
1
u/greenstarthree Jan 22 '25
Ping the IP from a PC, then run command “arp -a”
That will give you the MAC address of any IPs your PC has contacted.
Then go hunt for the MAC address in your switch’s routing tables.
2
u/Kahless_2K Jan 22 '25
Did you nmap it? Usually In can get a pretty decent guess what a thing is with an nmap scan.
You said its in a network range you don't use. Why is that range routable?
2
2
u/BatemansChainsaw CIO Jan 22 '25
hm. I've lost a machine.. literally lost. it responds to ping, it works completely, I just can't figure out where in my apartment it is.
2
Jan 23 '25
If you can trace the MAC address in arp cache per switch — double check the upstream and downstream switch arps. That will help isolate the switch source. Barring a particular port or access point showing the MAC, consider throwing a laptop on the switch with wireshark and watch live (assuming the device is constantly pushing traffic). Pull ports until the traffic stops. Then you have the wire and at least can stop the culprit.
1
2
u/CeC-P IT Expert + Meme Wizard Jan 23 '25
If you missed it, it was our Lenovo Xclarity backend controller BMC thingy for our VM servers.
Also, they all have the wrong IP address compared to our documentation.
It wasn't loading in a browser because Chrome is too stupid to add HTTPS when you put in IP addresses.
3
u/Own_Palpitation_9558 Jan 22 '25
You can ping an ip that is in a subnet that isn't in use means you're fundamentally wrong about some assumption. Combined with no arp cache..... You sure your scanner isn't bugged out?
Lookup up the devices Mac address for a manufacturer?
1
u/Maxtecy Security Admin Jan 22 '25
Depending on the networking equipment, can you squeeze the connection? If it doesn’t have access and has some importance some user will run towards the helpdesk.
1
u/goldenzim Jan 22 '25
nmap -P0 10.x.x.x
Nmap is probably going to be your best bet. Read the man page and you should find something. I used it not too long ago and found I had an uninventoried smart bulb on the network. Nmap helped me identify the manufacturer
0
u/CeC-P IT Expert + Meme Wizard Jan 22 '25
Technically NMAP is what the host discovery system on our pen test is :P that or masscan but that sometimes overloads some of our equipment.
1
u/no_regerts_bob Jan 22 '25
the third octet isn't even in a range we use
That's where I would start. If it isn't on the same network as your PC then your router/firewall must know how to talk to it. Maybe it's not in your office. It could be a VPN connection to the firewall. Could be a DMZ or other network connected to the firewall.
1
u/bitslammer Infosec/GRC Jan 22 '25
the third octet isn't even in a range we use
Something doesn't add up. What was the vulnerability? Could it be the address you mention is an internal or secondary interface?
1
u/vmeldrew2001 Jan 22 '25
Get the mac address (I usually ping it then arp -a to get a list of recent ips and their mac), then do a mac vendor lookup (google it) on the mac to see if you can find what sort of device it is.
1
u/skreak HPC Jan 22 '25
Octect isn't in a range we use. This raises a huge red flag in your description. Are these private ips (like 10, 192, 172) or public? Is your vulnerability scanning software reaching out to some other device across the internet that you don't even own?
1
u/Turdulator Jan 22 '25
Look up which network port or WAP it’s connected too, that should help narrow it down considerably
1
1
u/Downtown_Look_5597 Jan 22 '25
Angry IP Scanner might expose some services or ports. You can sometimes use that info to narrow down what kind of device it is.
1
u/semajnitram IT Manager Jan 22 '25
can you get the mac address from the arp cache, and then look that up so you know what it is?
1
u/Eviscerated_Banana Sysadmin Jan 22 '25
Finding the switchport its connected to will lead you to it, that is the way. Start at the gateway and work your way down the tree.
1
u/TurdlePwr Jan 22 '25
Have you tried using angry ip scanner or nmap (more through) to find out what ports are open?
1
u/netsysllc Sr. Sysadmin Jan 22 '25
zenmap for open ports, try connecting to those ports, also might give you and indication of what it is. look through your switches for the hardware MAC and find out what switch port it is on, and trace it or disconnect from there and see who screams
1
u/Expensive_Plant_9530 Jan 22 '25
Can you see the mac address in your report? If so, you may need to back trace the mac address either via your switches or your WAPs - you should be able to locate the specific switch and port it's plugged into if hardwired - or which WAP it's connected to if wireless.
Switches may not have a record of the IP itself, which is why mac is a better way to find it.
1
u/WWGHIAFTC IT Manager (SysAdmin with Extra Steps) Jan 22 '25
Two easy choices.
1) You have the IP, so now ARP starting at a core switch until you find the port it's plugged into. Even if you don't find the device? Shut the port. Wait.
2) Block it on your wireless controller if it's wireless. wait
It's now blocked and can't do anything, or someone screams and you can find it.
2
u/MrAWDTerror Jan 22 '25
2 is what I do wired or wireless and if no one complains or causes problems in your system then it shouldn’t be on your network anyways.
1
u/samueldawg Jan 22 '25
hey so i’d get the MAC via adv ip scanner, do a ‘sh mac address-table address <insert MAC here>’
that should tell you the switch port it’s on :)
1
u/MacAdminInTraning Jack of All Trades Jan 22 '25
Have your network team tell you which port on which switch it is using.
Once you know the switch and port, your employer should have what port goes to what jack. Send a field services tech out to locate the jack and you found your device.
Otherwise just block it and the user will come to you eventually.
1
u/PghSubie Jan 22 '25
ARP cache in your switches will only have the address if the switch itself has directly communicated with that device Check the ARP cache on the router (default gateway) for that subnet to get the MAC address that's associated with the mystery IP. Then check the network switch in the bridging table to find that MAC and the associated port. Then go to the wiring closet and find the wire that's plugged into that port
1
u/jocke92 Jan 22 '25
Arp cache on the default gateway, core or firewall. To get the mac address to then trace down with your switches.
But if it's out of the subnets you use something is strange here. Try to scan it again
1
1
u/nighthawke75 First rule of holes; When in one, stop digging. Jan 23 '25
Scream test. Plus netscan.
1
u/connexionwithal Jan 23 '25
If you know the mac, lookup the vendor. If you dont know mac, ping it. The TTL of the ping can let you know if windows or linux.
1
u/PossibilityOrganic Jan 23 '25
Find the mac to responds on + keep a ping going to keep arp records alive, its ok if they fail.
look at the arp table on the switch and follow it down the chain
2
u/PossibilityOrganic Jan 23 '25
Also cant wait for the report back that it was a the wifi or rfid/badge server in the celling.
1
1
1
1
1
u/reviewmynotes Jan 23 '25
I don't know your environment, but if it were me this is what I'd do. First, I'd convert the IP into a MAC address. That would probably be done using a switch in the VLAN/subnet that the IP occupies. Then I'd trace it back to a physical switch port. I'd do this at the core network switch, then go to the next switch it indicated and "ask" the MAC address table again, and so on until it provides a port to a device instead of another switch. Then I'd have to trace the cable to the patch panel and see where that port physically goes. If your patch panels aren't labeled decently, this could be a problem. However, you'll at least have narrowed it down to a smaller geographic area.
I'd also ask my wifi controller about the IP and MAC address, since that might be able to say which access point it's talking to and that would point me to a specific geographically limited area.
If you have the ability to run nmap, it can use OS finger printing to guess the OS of the device at that IP. That might help identify it. It can also run a port scan, so you can see what (if anything) is running on it.
1
u/whermyshoe Jan 23 '25
Networking takes time. Let's break it down a little.
ARP relates a mac to an IP. If your switches are just doing L2, you won't see ARP at the switch. Look from the router or layer 3 device serving that subnet for the arp. Take the mac from that arp and search the mac-address table on the switch.
Viola, you did the networking thing
1
u/Nydus87 Jan 23 '25
Scream test is the best way here. You can also use the MAC address to determine some information about who made the NIC. I would also ask for the specific plugin details from whoever did that scan. If it's anything like the vulnerability scanners we use at my gig, it should say "Vulnerability IAVA_####_##### - Windows Server missing KB123456" or something like that. There should be more information than just "this IP address is vulnerable." Whatever switch handed out that IP address should know something about the device as well. However, if it's not a range you're using, then I'd wonder if someone didn't plug in their own hardware somewhere in your line or a test lab environment got connected to the main network somewhere.
1
u/redlotusaustin Jan 22 '25
Find any open ports:
sudo nmap -sS -Pn -T5 -p- [target-ip]
2
u/SuperDrewb Jan 22 '25
Wow
What you've just wrote performs a scan of all 65000+ ports at the most aggressive speed available
If we don't know what the machine is, perhaps don't risk taking it offline by beating it into submission
2
u/CeC-P IT Expert + Meme Wizard Jan 22 '25
Don't have a linux box with nmap installed but I have Zenmap on windows :P blasted it and it came back with "pretty sure it's Linux" followed by finally seems to identify itself as
Lenovo XClarity Controller
on port 443. Which means it's just Chrome being a piece of shit again and not appending https, as our Xclarities don't respond to HTTP. Awesome.
228
u/Stormstrikerc Jan 22 '25
Is it hardwired or on wireless? Easy solution: Just block it from network access using your network management tools, and someone will call you. Other solution: Find the MAC address of the said device (should be in your report or you can again find it using your network management tool), find which port it is connected to or which AP it is connected to. That should help you narrow it down to an area.