Skip to content

Dynamic Database Configuration with Automatic Failover

Eliminate Hardcoded IPs - Enable Easy Database Switching

Date: 2025-11-12 Problem: 15+ hardcoded database IPs across appsettings.json Solution: Dynamic configuration with health checks and automatic failover


πŸ”΄ CURRENT PROBLEM

Hardcoded IPs Found:

SQL Server Databases (10+ different servers):
β”œβ”€ 10.32.8.130:1988 (Primary)
β”œβ”€ 10.32.8.140:1988 (Supplier logs)
β”œβ”€ 10.32.8.152:1433 (Log tracking)
β”œβ”€ 10.32.8.142 (Cache manager)
β”œβ”€ 10.32.8.143:1433 (Replication DB1)
β”œβ”€ 10.32.8.149:1433 (Replication DB2)
β”œβ”€ 10.32.8.9:1433 (Replication DB3)
β”œβ”€ 10.32.8.5:1433 (Replication DB)
β”œβ”€ 10.32.8.85:1433 (Replication DB4)
β”œβ”€ 10.32.8.11 (OTH Replica)
β”œβ”€ 10.32.8.105 (OTH2 Replica)
β”œβ”€ 10.32.8.34 (CM2)
β”œβ”€ 10.32.8.16:1433 (Log server2)
└─ 10.32.8.180:1433 (Log track 180VM)

MongoDB Databases (8+ servers):
β”œβ”€ 10.32.8.51:27017 (Cache 1)
β”œβ”€ 10.32.8.52:27017 (Cache 2)
β”œβ”€ 10.32.8.53:27017 (Cache 3)
β”œβ”€ 10.32.8.18:27017 (Supplier logs)
β”œβ”€ 10.32.8.96:27017 (Room mapping)
β”œβ”€ 10.32.8.101:27017 (Unmapped rooms)
β”œβ”€ 10.32.8.75:27017 (API RQ/RS logs)
└─ 10.32.8.74:27017 (Supplier performance)

Redis:
└─ 10.32.8.205:6379 (Cache)

RabbitMQ:
└─ 10.32.8.90:5672 (Message queue)

Issues: - ❌ If ANY database goes down, manual appsettings.json edit needed - ❌ Requires application restart to change endpoints - ❌ No automatic failover - ❌ Hard to manage 20+ IPs - ❌ Deployment nightmare (different IPs per environment)


βœ… SOLUTION: 3-TIER APPROACH

Tier 1: External Configuration (BEST) ⭐

Use external config service (Consul, etcd, or Azure App Configuration)

Tier 2: Database-Driven Configuration (GOOD)

Store all endpoints in a master config database

Tier 3: Enhanced appsettings.json (OK)

Use DNS names + failover lists in appsettings


Using HashiCorp Consul (FREE, Open-Source)

Architecture:

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚    XConnect Application              β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
         β”‚
         β–Ό
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚    Consul (Config + Service Discovery)β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚  PRIMARY_DB: sql01.withinearth.local β”‚
β”‚  FAILOVER_DB: sql02.withinearth.localβ”‚
β”‚  MONGODB: mongo-cluster.local        β”‚
β”‚  REDIS: redis-cluster.local          β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
         β”‚
         β–Ό
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚    Health Checks (Every 10 seconds)  β”‚
β”‚  βœ“ Check if sql01 is UP              β”‚
β”‚  βœ“ If DOWN β†’ Switch to sql02         β”‚
β”‚  βœ“ Auto-update configuration         β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜


Step 1: Install Consul

# On your config server (can be monitoring server)
cd /home/monitor

# Download Consul
wget https://releases.hashicorp.com/consul/1.17.0/consul_1.17.0_linux_amd64.zip
unzip consul_1.17.0_linux_amd64.zip
sudo mv consul /usr/local/bin/

# Create Consul config
sudo mkdir -p /etc/consul.d
sudo tee /etc/consul.d/config.json << 'EOF'
{
  "datacenter": "withinearth-dc1",
  "data_dir": "/opt/consul",
  "log_level": "INFO",
  "server": true,
  "bootstrap_expect": 1,
  "ui": true,
  "client_addr": "0.0.0.0",
  "bind_addr": "0.0.0.0",
  "ports": {
    "http": 8500,
    "dns": 8600
  }
}
EOF

# Create systemd service
sudo tee /etc/systemd/system/consul.service << 'EOF'
[Unit]
Description=Consul
Documentation=https://www.consul.io/
After=network-online.target
Wants=network-online.target

[Service]
User=root
Group=root
ExecStart=/usr/local/bin/consul agent -config-dir=/etc/consul.d/
ExecReload=/bin/kill -HUP $MAINPID
KillMode=process
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF

# Start Consul
sudo systemctl enable consul
sudo systemctl start consul

# Check status
consul members

# Access UI
# http://[server-ip]:8500/ui

Step 2: Configure Database Endpoints in Consul

# Add primary database config
consul kv put config/withinearth/databases/primary/host "10.32.8.130"
consul kv put config/withinearth/databases/primary/port "1988"
consul kv put config/withinearth/databases/primary/database "withinearthUpdated"
consul kv put config/withinearth/databases/primary/user "sa"
consul kv put config/withinearth/databases/primary/password "FG#\$2adfgk%^we981988"

# Add failover databases
consul kv put config/withinearth/databases/failover1/host "10.32.8.5"
consul kv put config/withinearth/databases/failover1/port "1433"

consul kv put config/withinearth/databases/failover2/host "10.32.8.143"
consul kv put config/withinearth/databases/failover2/port "1433"

# Add MongoDB cluster
consul kv put config/withinearth/mongodb/hosts "10.32.8.51:27017,10.32.8.52:27017,10.32.8.53:27017"
consul kv put config/withinearth/mongodb/database "withinearth"

# Add Redis
consul kv put config/withinearth/redis/host "10.32.8.205"
consul kv put config/withinearth/redis/port "6379"

# Add RabbitMQ
consul kv put config/withinearth/rabbitmq/host "10.32.8.90"
consul kv put config/withinearth/rabbitmq/port "5672"

Step 3: Add Health Checks

# Create health check config
sudo tee /etc/consul.d/health-checks.json << 'EOF'
{
  "checks": [
    {
      "id": "primary-sql-server",
      "name": "Primary SQL Server Health",
      "tcp": "10.32.8.130:1988",
      "interval": "10s",
      "timeout": "2s"
    },
    {
      "id": "mongodb-node1",
      "name": "MongoDB Node 1 Health",
      "tcp": "10.32.8.51:27017",
      "interval": "10s",
      "timeout": "2s"
    },
    {
      "id": "redis-server",
      "name": "Redis Health",
      "tcp": "10.32.8.205:6379",
      "interval": "10s",
      "timeout": "2s"
    },
    {
      "id": "rabbitmq-server",
      "name": "RabbitMQ Health",
      "tcp": "10.32.8.90:5672",
      "interval": "10s",
      "timeout": "2s"
    }
  ]
}
EOF

# Reload Consul
sudo systemctl reload consul

Step 4: Update .NET Application to Use Consul

Install NuGet Package:

dotnet add package Consul
dotnet add package Microsoft.Extensions.Configuration.Consul

Create ConsulConfigurationProvider.cs:

using Consul;
using Microsoft.Extensions.Configuration;

namespace XConnect.Configuration
{
    public class ConsulConfigurationProvider : ConfigurationProvider
    {
        private readonly ConsulClient _consulClient;
        private readonly string _consulAddress;
        private readonly string _prefix;
        private Timer? _refreshTimer;

        public ConsulConfigurationProvider(string consulAddress, string prefix = "config/withinearth")
        {
            _consulAddress = consulAddress;
            _prefix = prefix;
            _consulClient = new ConsulClient(config => {
                config.Address = new Uri(consulAddress);
            });
        }

        public override void Load()
        {
            LoadFromConsul().GetAwaiter().GetResult();

            // Refresh every 30 seconds
            _refreshTimer = new Timer(
                _ => LoadFromConsul().GetAwaiter().GetResult(),
                null,
                TimeSpan.FromSeconds(30),
                TimeSpan.FromSeconds(30)
            );
        }

        private async Task LoadFromConsul()
        {
            try
            {
                var result = await _consulClient.KV.List(_prefix);

                if (result?.Response != null)
                {
                    var data = new Dictionary<string, string?>(StringComparer.OrdinalIgnoreCase);

                    foreach (var kvPair in result.Response)
                    {
                        var key = kvPair.Key
                            .Substring(_prefix.Length + 1)  // Remove prefix
                            .Replace("/", ":");              // Use : separator

                        var value = System.Text.Encoding.UTF8.GetString(kvPair.Value ?? Array.Empty<byte>());
                        data[key] = value;
                    }

                    Data = data;
                    OnReload();  // Notify that config changed
                }
            }
            catch (Exception ex)
            {
                Console.WriteLine($"Error loading config from Consul: {ex.Message}");
            }
        }
    }

    public class ConsulConfigurationSource : IConfigurationSource
    {
        public string ConsulAddress { get; set; } = "http://localhost:8500";
        public string Prefix { get; set; } = "config/withinearth";

        public IConfigurationProvider Build(IConfigurationBuilder builder)
        {
            return new ConsulConfigurationProvider(ConsulAddress, Prefix);
        }
    }

    public static class ConsulConfigurationExtensions
    {
        public static IConfigurationBuilder AddConsul(
            this IConfigurationBuilder builder,
            string consulAddress,
            string prefix = "config/withinearth")
        {
            return builder.Add(new ConsulConfigurationSource
            {
                ConsulAddress = consulAddress,
                Prefix = prefix
            });
        }
    }
}

Update Program.cs:

using XConnect.Configuration;

var builder = WebApplication.CreateBuilder(args);

// IMPORTANT: Add Consul BEFORE appsettings.json
// This way Consul values override static config
builder.Configuration
    .AddJsonFile("appsettings.json", optional: false, reloadOnChange: true)
    .AddConsul("http://10.32.8.209:8500", "config/withinearth")  // ← Add Consul
    .AddEnvironmentVariables();

// Now services can use IConfiguration normally
builder.Services.AddDbContext<AppDbContext>(options =>
{
    // This will read from Consul automatically!
    var host = builder.Configuration["databases:primary:host"];
    var port = builder.Configuration["databases:primary:port"];
    var database = builder.Configuration["databases:primary:database"];
    var user = builder.Configuration["databases:primary:user"];
    var password = builder.Configuration["databases:primary:password"];

    var connectionString = $"Data Source={host},{port};Initial Catalog={database};User ID={user};Password={password};TrustServerCertificate=True;";

    options.UseSqlServer(connectionString);
});

var app = builder.Build();
app.Run();

Benefits: - βœ… Change database IP in Consul β†’ Application auto-reloads in 30 seconds - βœ… No application restart needed - βœ… Health checks β†’ Auto-failover if primary DB down - βœ… Version control β†’ Track config changes - βœ… Multi-environment β†’ Different configs for dev/staging/prod - βœ… UI Dashboard β†’ View/edit configs in browser


🎯 TIER 2: DATABASE-DRIVEN CONFIGURATION (SIMPLER)

Store all endpoints in a master configuration database.

Step 1: Create ConfigMaster Database

-- Create on a stable server (e.g., 10.32.8.130)
CREATE DATABASE ConfigMaster;
GO

USE ConfigMaster;
GO

CREATE TABLE DatabaseEndpoints (
    Id INT IDENTITY(1,1) PRIMARY KEY,
    EnvironmentName VARCHAR(50) NOT NULL,  -- Production, Staging, Dev
    EndpointKey VARCHAR(100) NOT NULL,      -- PRIMARY_DB, MONGO_CACHE, etc.
    EndpointType VARCHAR(50) NOT NULL,      -- SQL, MongoDB, Redis, RabbitMQ
    HostName VARCHAR(255) NOT NULL,
    Port INT NOT NULL,
    DatabaseName VARCHAR(100),
    Username VARCHAR(100),
    PasswordEncrypted VARBINARY(MAX),       -- Encrypted password
    Priority INT DEFAULT 1,                  -- 1=Primary, 2=Secondary, etc.
    IsActive BIT DEFAULT 1,
    LastHealthCheck DATETIME,
    HealthStatus VARCHAR(20),               -- Healthy, Degraded, Down
    CreatedDate DATETIME DEFAULT GETDATE(),
    ModifiedDate DATETIME DEFAULT GETDATE(),
    CONSTRAINT UK_Endpoint UNIQUE (EnvironmentName, EndpointKey, Priority)
);
GO

-- Insert current endpoints
INSERT INTO DatabaseEndpoints (EnvironmentName, EndpointKey, EndpointType, HostName, Port, DatabaseName, Username, Priority, IsActive)
VALUES
-- Primary SQL Server
('Production', 'PRIMARY_DB', 'SQL', '10.32.8.130', 1988, 'withinearthUpdated', 'sa', 1, 1),
('Production', 'PRIMARY_DB', 'SQL', '10.32.8.5', 1433, 'withinearthUpdated', 'apirep', 2, 1),  -- Failover
('Production', 'PRIMARY_DB', 'SQL', '10.32.8.143', 1433, 'withinearthUpdated', 'sa', 3, 1),     -- Failover 2

-- MongoDB cluster
('Production', 'MONGO_CACHE', 'MongoDB', '10.32.8.51', 27017, 'withinearth', NULL, 1, 1),
('Production', 'MONGO_CACHE', 'MongoDB', '10.32.8.52', 27017, 'withinearth', NULL, 2, 1),
('Production', 'MONGO_CACHE', 'MongoDB', '10.32.8.53', 27017, 'withinearth', NULL, 3, 1),

-- Supplier logs MongoDB
('Production', 'MONGO_SUPPLIER_LOG', 'MongoDB', '10.32.8.18', 27017, 'SupplierLogs', NULL, 1, 1),

-- Redis
('Production', 'REDIS_CACHE', 'Redis', '10.32.8.205', 6379, NULL, NULL, 1, 1),

-- RabbitMQ
('Production', 'RABBITMQ', 'RabbitMQ', '10.32.8.90', 5672, NULL, 'admin', 1, 1);
GO

-- Create stored procedure to get active endpoint
CREATE PROCEDURE sp_GetActiveEndpoint
    @Environment VARCHAR(50),
    @EndpointKey VARCHAR(100)
AS
BEGIN
    SELECT TOP 1
        HostName,
        Port,
        DatabaseName,
        Username,
        CAST(DECRYPTBYPASSPHRASE('YourEncryptionKey', PasswordEncrypted) AS VARCHAR(100)) AS Password,
        Priority,
        HealthStatus
    FROM DatabaseEndpoints
    WHERE EnvironmentName = @Environment
        AND EndpointKey = @EndpointKey
        AND IsActive = 1
        AND (HealthStatus IS NULL OR HealthStatus = 'Healthy')
    ORDER BY Priority ASC;
END
GO

-- Create health check procedure
CREATE PROCEDURE sp_UpdateEndpointHealth
    @EndpointKey VARCHAR(100),
    @HealthStatus VARCHAR(20)
AS
BEGIN
    UPDATE DatabaseEndpoints
    SET HealthStatus = @HealthStatus,
        LastHealthCheck = GETDATE()
    WHERE EndpointKey = @EndpointKey
        AND EnvironmentName = 'Production';
END
GO

Step 2: Create Configuration Service in .NET

using System.Data.SqlClient;
using Microsoft.Extensions.Caching.Memory;

namespace XConnect.Configuration
{
    public interface IEndpointConfigService
    {
        Task<DatabaseEndpoint?> GetActiveEndpointAsync(string endpointKey);
        Task UpdateHealthStatusAsync(string endpointKey, string healthStatus);
        string BuildConnectionString(DatabaseEndpoint endpoint);
    }

    public class DatabaseEndpoint
    {
        public string HostName { get; set; } = "";
        public int Port { get; set; }
        public string? DatabaseName { get; set; }
        public string? Username { get; set; }
        public string? Password { get; set; }
        public int Priority { get; set; }
        public string? HealthStatus { get; set; }
    }

    public class EndpointConfigService : IEndpointConfigService
    {
        private readonly string _configDbConnectionString;
        private readonly IMemoryCache _cache;
        private readonly ILogger<EndpointConfigService> _logger;
        private const string ENVIRONMENT = "Production";

        public EndpointConfigService(
            IConfiguration configuration,
            IMemoryCache cache,
            ILogger<EndpointConfigService> logger)
        {
            // Config DB is the only hardcoded connection (stable server)
            _configDbConnectionString = configuration["ConfigDb:ConnectionString"]
                ?? "Data Source=10.32.8.130,1988;Initial Catalog=ConfigMaster;User ID=config_reader;Password=SecurePass123!;TrustServerCertificate=True;";

            _cache = cache;
            _logger = logger;
        }

        public async Task<DatabaseEndpoint?> GetActiveEndpointAsync(string endpointKey)
        {
            // Check cache first (1 minute TTL)
            var cacheKey = $"endpoint_{endpointKey}";
            if (_cache.TryGetValue<DatabaseEndpoint>(cacheKey, out var cachedEndpoint))
            {
                return cachedEndpoint;
            }

            // Fetch from config database
            try
            {
                using var connection = new SqlConnection(_configDbConnectionString);
                await connection.OpenAsync();

                using var command = new SqlCommand("sp_GetActiveEndpoint", connection);
                command.CommandType = System.Data.CommandType.StoredProcedure;
                command.Parameters.AddWithValue("@Environment", ENVIRONMENT);
                command.Parameters.AddWithValue("@EndpointKey", endpointKey);

                using var reader = await command.ExecuteReaderAsync();

                if (await reader.ReadAsync())
                {
                    var endpoint = new DatabaseEndpoint
                    {
                        HostName = reader.GetString(0),
                        Port = reader.GetInt32(1),
                        DatabaseName = reader.IsDBNull(2) ? null : reader.GetString(2),
                        Username = reader.IsDBNull(3) ? null : reader.GetString(3),
                        Password = reader.IsDBNull(4) ? null : reader.GetString(4),
                        Priority = reader.GetInt32(5),
                        HealthStatus = reader.IsDBNull(6) ? null : reader.GetString(6)
                    };

                    // Cache for 1 minute
                    _cache.Set(cacheKey, endpoint, TimeSpan.FromMinutes(1));

                    return endpoint;
                }

                _logger.LogWarning($"No active endpoint found for key: {endpointKey}");
                return null;
            }
            catch (Exception ex)
            {
                _logger.LogError(ex, $"Error fetching endpoint config for {endpointKey}");
                throw;
            }
        }

        public async Task UpdateHealthStatusAsync(string endpointKey, string healthStatus)
        {
            try
            {
                using var connection = new SqlConnection(_configDbConnectionString);
                await connection.OpenAsync();

                using var command = new SqlCommand("sp_UpdateEndpointHealth", connection);
                command.CommandType = System.Data.CommandType.StoredProcedure;
                command.Parameters.AddWithValue("@EndpointKey", endpointKey);
                command.Parameters.AddWithValue("@HealthStatus", healthStatus);

                await command.ExecuteNonQueryAsync();

                // Invalidate cache
                _cache.Remove($"endpoint_{endpointKey}");
            }
            catch (Exception ex)
            {
                _logger.LogError(ex, $"Error updating health status for {endpointKey}");
            }
        }

        public string BuildConnectionString(DatabaseEndpoint endpoint)
        {
            if (endpoint.DatabaseName != null)
            {
                // SQL Server connection string
                return $"Data Source={endpoint.HostName},{endpoint.Port};" +
                       $"Initial Catalog={endpoint.DatabaseName};" +
                       $"User ID={endpoint.Username};Password={endpoint.Password};" +
                       $"TrustServerCertificate=True;";
            }
            else
            {
                // For Redis, MongoDB, etc - return simple host:port
                return $"{endpoint.HostName}:{endpoint.Port}";
            }
        }
    }
}

Step 3: Use Dynamic Configuration in Your Application

// Program.cs
builder.Services.AddMemoryCache();
builder.Services.AddSingleton<IEndpointConfigService, EndpointConfigService>();

// Use dynamic connection
builder.Services.AddDbContext<AppDbContext>(async (services, options) =>
{
    var configService = services.GetRequiredService<IEndpointConfigService>();
    var endpoint = await configService.GetActiveEndpointAsync("PRIMARY_DB");

    if (endpoint == null)
    {
        throw new Exception("Primary database endpoint not configured!");
    }

    var connectionString = configService.BuildConnectionString(endpoint);
    options.UseSqlServer(connectionString);
});


// In your repository or service
public class BookingRepository
{
    private readonly IEndpointConfigService _configService;

    public BookingRepository(IEndpointConfigService configService)
    {
        _configService = configService;
    }

    public async Task<Booking?> GetBookingAsync(int bookingId)
    {
        // Get active endpoint dynamically
        var endpoint = await _configService.GetActiveEndpointAsync("PRIMARY_DB");
        var connectionString = _configService.BuildConnectionString(endpoint!);

        using var connection = new SqlConnection(connectionString);
        await connection.OpenAsync();

        // Your query here...
        return await connection.QueryFirstOrDefaultAsync<Booking>(
            "SELECT * FROM OnlineHotelBooking WHERE BookingId = @Id",
            new { Id = bookingId }
        );
    }
}

Step 4: Add Background Health Check Service

public class DatabaseHealthCheckService : BackgroundService
{
    private readonly IEndpointConfigService _configService;
    private readonly ILogger<DatabaseHealthCheckService> _logger;

    public DatabaseHealthCheckService(
        IEndpointConfigService configService,
        ILogger<DatabaseHealthCheckService> logger)
    {
        _configService = configService;
        _logger = logger;
    }

    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        while (!stoppingToken.IsCancellationRequested)
        {
            try
            {
                await CheckEndpointHealth("PRIMARY_DB");
                await CheckEndpointHealth("MONGO_CACHE");
                await CheckEndpointHealth("REDIS_CACHE");
                await CheckEndpointHealth("RABBITMQ");

                // Check every 30 seconds
                await Task.Delay(TimeSpan.FromSeconds(30), stoppingToken);
            }
            catch (Exception ex)
            {
                _logger.LogError(ex, "Error in health check service");
            }
        }
    }

    private async Task CheckEndpointHealth(string endpointKey)
    {
        try
        {
            var endpoint = await _configService.GetActiveEndpointAsync(endpointKey);
            if (endpoint == null) return;

            var isHealthy = await PingEndpoint(endpoint);

            await _configService.UpdateHealthStatusAsync(
                endpointKey,
                isHealthy ? "Healthy" : "Down"
            );

            if (!isHealthy)
            {
                _logger.LogWarning($"Endpoint {endpointKey} is DOWN! Auto-failover will occur.");
            }
        }
        catch (Exception ex)
        {
            _logger.LogError(ex, $"Error checking health for {endpointKey}");
        }
    }

    private async Task<bool> PingEndpoint(DatabaseEndpoint endpoint)
    {
        try
        {
            using var client = new System.Net.Sockets.TcpClient();
            await client.ConnectAsync(endpoint.HostName, endpoint.Port);
            return client.Connected;
        }
        catch
        {
            return false;
        }
    }
}

// Register in Program.cs
builder.Services.AddHostedService<DatabaseHealthCheckService>();

🎯 TIER 3: DNS-BASED APPROACH (SIMPLEST)

Use DNS names instead of IPs.

Step 1: Setup DNS (Internal DNS Server)

# On your DNS server or /etc/hosts on all API servers

# Add DNS entries
sudo tee -a /etc/hosts << 'EOF'
# Primary databases
10.32.8.130    sql-primary.withinearth.local
10.32.8.5      sql-failover1.withinearth.local
10.32.8.143    sql-failover2.withinearth.local

# MongoDB cluster
10.32.8.51     mongo-node1.withinearth.local
10.32.8.52     mongo-node2.withinearth.local
10.32.8.53     mongo-node3.withinearth.local

# Redis
10.32.8.205    redis-primary.withinearth.local

# RabbitMQ
10.32.8.90     rabbitmq-primary.withinearth.local
EOF

Step 2: Update appsettings.json with DNS Names

{
  "ConnectionStrings": {
    "ConnectionString": "Data Source=sql-primary.withinearth.local,1988;Initial Catalog=withinearthUpdated;User ID=sa;Password=FG#$2adfgk%^we981988;TrustServerCertificate=True;",

    "ConnectionString_Failovers": [
      "Data Source=sql-failover1.withinearth.local,1433;Initial Catalog=withinearthUpdated;User ID=apirep;Password=c2fH#B3yJJfcDf3U;TrustServerCertificate=True;",
      "Data Source=sql-failover2.withinearth.local,1433;Initial Catalog=withinearthUpdated;User ID=sa;Password=U76&j$v7ni5y*LG5;TrustServerCertificate=True;"
    ]
  },

  "MongoSettings": {
    "MongoDB_ConnectionStr": "mongodb://mongo-node1.withinearth.local:27017,mongo-node2.withinearth.local:27017,mongo-node3.withinearth.local:27017/?replicaSet=rs0",
    "MongoDB_DatabaseName": "withinearth"
  },

  "RedisSettings": {
    "RedisDB_ConnectionStr1": "redis-primary.withinearth.local:6379"
  },

  "RabitMQSettings": {
    "HostName": "rabbitmq-primary.withinearth.local",
    "Port": "5672"
  }
}

Benefits: - βœ… Change IP by updating DNS (no app restart!) - βœ… Simple to implement - βœ… Works immediately

Cons: - ⚠️ DNS caching (may take 5-60 seconds to propagate) - ⚠️ No automatic health checks


πŸ“Š COMPARISON TABLE

Solution Complexity Failover No Restart Cost Best For
Consul High βœ… Auto βœ… Yes Free Large scale, cloud
Database Config Medium βœ… Auto βœ… Yes Free Current setup
DNS Low ⚠️ Manual βœ… Yes Free Quick fix
Hardcoded IPs Low ❌ No ❌ No Free NOT recommended

Phase 1: Quick Win (This Week) - DNS

  1. Add DNS entries to /etc/hosts
  2. Replace IPs with DNS names in appsettings.json
  3. Test and deploy

Effort: 2 hours Benefit: Easy IP changes without code edits


Phase 2: Proper Solution (Next Month) - Database Config

  1. Create ConfigMaster database
  2. Implement EndpointConfigService
  3. Add health check background service
  4. Migrate all endpoints to database

Effort: 1 week Benefit: Auto-failover, centralized management


Phase 3: Enterprise (Optional) - Consul

  1. Setup Consul cluster
  2. Migrate to Consul configuration
  3. Add service discovery

Effort: 2 weeks Benefit: Cloud-ready, full observability


βœ… NEXT STEPS

Want me to: 1. Create the ConfigMaster database scripts? 2. Generate the full .NET configuration service code? 3. Setup DNS entries for your servers? 4. Create deployment scripts?

Just let me know which tier you want to implement! πŸš€