Standard Operating Procedure (SOP) for Troubleshooting Issues in your SafeSquid proxy server

Purpose:

To provide a consistent and effective approach to troubleshooting network connectivity and web server issues using key commands.

Scope:

This SOP applies to troubleshooting sessions involving DNS resolution, port connectivity, web server certificates, and web server responses, with and without proxy usage.

Procedure:

Verify DNS Resolution:

Use the command nslookup to check if the domain name can be resolved to an IP address.

nslookup <hostname>

Successful scenario: Screenshot of successful nslookup output.

nslookup safesquid.com

image

Unsuccessful scenario: Screenshot of nslookup output indicating resolution failure.

nslookup safesquid.com

image

If resolution fails, check DNS settings, server availability, and firewall rules.

Check Port Connectivity:

Use the command telnet to attempt a connection to port 443 on the server.

 echo "" | telnet <hostname> 443

Successful scenario: Screenshot of successful telnet connection.

 echo "" | telnet safesquid.com 443

image

Unsuccessful scenario: Screenshot of telnet connection failure.

 echo "" | telnet safesquid.com 443

image

If unsuccessful, verify firewall rules, network restrictions, and server configuration.

Test Web Server Certificate (if required/ if the issue is related to server certificates):

Use the command openssl s_client to examine the server’s certificate details.

echo "" | openssl s_client -brief  -state -connect <hostname>:443  -verify_hostname <hostname>

Successful scenario: Screenshot of valid certificate details from openssl s_client.

echo "" | openssl s_client -brief  -state -connect safesquid.com:443  -verify_hostname safesquid.com

image

Unsuccessful scenario: Screenshot of invalid or expired certificate details.

echo "" | openssl s_client -brief  -state -connect incomplete-chain.badssl.com:443  -verify_hostname incomplete-chain.badssl.com

image

Check for validity, expiration, and compatibility with client requirements.

Check Web Server Response:

Use the curl command to retrieve a response from the web server.

curl https://<hostname> --head 

Successful scenario: Screenshot of successful curl output with a 200-status code.

curl https://safesquid.com --head

curl https://157.245.103.186 --head

Unsuccessful scenario: Screenshot of curl output.
image

Analyze response codes (e.g., 200 for success, 404 for not found), headers, and content for troubleshooting purposes.

Test Response via Proxy:

Verify connectivity and web server response through the proxy.

Configure curl to use the proxy with the --proxy option to check the response via proxy.

curl --proxy 127.0.0.1:8080 --cacert /usr/local/safesquid/security/ssl/ROOT_X509File.cer  https://<hostname> --head

Successful scenario: Screenshot of successful curl output with proxy enabled.

curl --proxy 127.0.0.1:8080 --cacert /usr/local/safesquid/security/ssl/ROOT_X509File.cer  https://safesquid.com --head

Unsuccessful scenario: Screenshot of curl output with proxy blocking.

curl --proxy 127.0.0.1:8080 --cacert /usr/local/safesquid/security/ssl/ROOT_X509File.cer  https://safesquid.com --head

image

Access and Filter SafeSquid’s Extended Logs:

Use awk command to view a summary of connections related to the websites you are trying to debug for example “safesquid.com”:

awk -F'\t' '{print $4, $2, $11, $12, $6, $21, $24, $23, $33, $37}' /var/log/safesquid/extended/extended.log | grep "<website>"

Example:

awk -F'\t' '{print $4, $2, $11, $12, $6, $21, $15, $24, $23, $33, $37}' /var/log/safesquid/extended/extended.log | grep "safesquid.com"

Customize the output with different log keys as needed.

Interpret the Output:

Each line represents a connection, providing key details:

  • Date and time
  • Client IP
  • Username
  • Status code
  • Referrer
  • Requested host.
  • Peer
  • Categories
  • Profiles

Check for status code which ranges from 400-500.
Requests which are blocked by SafeSquid will have status code of 451.

You can get more specific by mentioning IP or username to get user specific logs.

awk -F'\t' '{print $4, $2, $11, $12, $6, $21, $24, $23, $33, $37}' /var/log/safesquid/extended/extended.log | grep "<WEBSITE>" | grep "<USER IP>"

Example:

 awk -F'\t' '{print $4, $2, $11, $12, $6, $21, $24, $23, $33, $37}' /var/log/safesquid/extended/extended.log | grep "safesquid.com" | grep "192.168.2.108"

This will only show traffic to the proxy generated from the specified user to the specified website.

image

Available Log Keys:

  1. record_id
  2. client_id
  3. request_id
  4. date_time
  5. elapsed_time
  6. status
  7. size
  8. upload
  9. download
  10. bypassed
  11. client_ip
  12. username
  13. method
  14. url
  15. http_referer
  16. useragent
  17. mime
  18. filter_name
  19. filtering_reason
  20. interface
  21. cachecode
  22. peercode
  23. peer
  24. request_host
  25. request_tld
  26. referer_host
  27. referer_tld
  28. range
  29. time_profiles
  30. user_groups
  31. request_profiles
  32. application_signatures
  33. categories
  34. response_profiles
  35. upload_content_types
  36. download_content_types
  37. profiles

Look for status code of 400-500:

Focus on Connections with 400-500 status codes these often indicate blocks or errors.

Use find_client_id.sh for Detailed Logs:

Note: Ensure that you have enabled debugging logs, you can find below on how to enable debugging log.

To retrieve complete logs for a specific connection, use find_client_id.sh script.

find_client_id.sh <client_id> | less

image

Check for any errors in the given connection.

Consult SafeSquid Support (If Necessary):

If you’re unable to resolve the issue after following the troubleshooting steps, contact SafeSquid support for further assistance.

Before contacting support, create a text file containing the detailed logs for a specific connection using the following command:

find_client_id.sh <client_id> > /home/administrator/error_<client_id>.txt

Replace <client_id> with the actual client ID.

Example:

find_client_id.sh 1257 > /home/administrator/error_1257.txt

image

Attach this text file to your support ticket to provide the technicians with comprehensive information for better troubleshooting.

Unable to find the root cause of your problem.

After trying all the mentioned troubleshooting steps, if you are still not able to figure out and solve your problem then consult SafeSquid’s support specialist with the output of your find_client_id.sh attached in your email.

To save the output of find_client_id.sh redirect the output to a text file as shown below.

find_client_id.sh 1257 > /home/administrator/error_1257.txt

When sending a support ticket for your issue add the file error\1257.txt for the support technician to support you better.

Troubleshooting Tips:

Identify Blocks: Look for blocking messages in the logs.

Status code 451 indicates that the website is getting blocked due to proxy rule.

Check Configurations: Review proxy rules and settings for potential misconfigurations.

Analyze Detailed Logs: Use find_client_id.sh to pinpoint specific issues.

Seek Further Assistance: If you need more help, consult SafeSquid documentation or contact their support team.

Remember:

Always enable debugging mode before troubleshooting.

Adapt the commands and log keys to your specific needs.

Consult SafeSquid documentation for more advanced troubleshooting.

If you are unable to identify the problem, share the output of find_client_id.sh with SafeSquid’s Support team at support@safesquid.net

Hey pratik thanks for the SOP.
This SOP had help us troubleshoot the issue very easily