Generate Sitemaps In Laravel

Generate Sitemaps In Laravel

A sitemap is an important part of any website. It stores all the URLs available on your website, usually in an HTML format or more commonly in an XML format. It allows your website to be accessed by web crawlers and retrieve necessary information using web scraping. Sitemaps can also boost your website’s SEO ranking since they make it easier for search engines to navigate through your website. They can be categorized into static and dynamic sitemaps and can be generated for both static and dynamic websites.

Sitemaps For Static Websites

Static websites contain hardcoded URLs and a fixed number of web pages. A sitemap for a static website can be created easily using an online sitemap generator tool such as XML-Sitemaps  (http://www.xml-sitemaps.com).

To use a sitemap generator, you need to enter your website’s URL to automatically generate a sitemap. Here, we are creating a static sitemap that can only be altered manually by a developer. This method works best for websites with a smaller yet fixed number of web pages. An example of a static sitemap would be as follows

<?xml version=“1.0” encoding=“UTF-8”?>
<urlset xmlns=“http://www.sitemaps.org/schemas/sitemap/0.9”>
    <url>
        <loc>https://www.example.com/</loc>
        <lastmod>2024-01-12T00:00:00+00:00</lastmod>
        <changefreq>weekly</changefreq>
        <priority>1.0</priority>
    </url>
    <url>
        <loc>https://www.example.com/about-us</loc>
        <lastmod>2024-01-12T00:00:00+00:00</lastmod>
        <changefreq>monthly</changefreq>
        <priority>0.8</priority>
    </url>
    <url>
        <loc>https://www.example.com/contact</loc>
        <lastmod>2024-01-12T00:00:00+00:00</lastmod>
        <changefreq>monthly</changefreq>
        <priority>0.7</priority>
    </url>
</urlset>

 

We observe that this sitemap contains the URLs of the ‘home’ page, the ‘about’ page, and the ‘contact’ page.

Seemingly static websites created by WordPress make use of plugins etc to generate sitemaps automatically. 

Sitemaps for Dynamic Websites

The code showed here is an actual example from NSE INDIA holiday API Portal

Generating a sitemap for a dynamic website is relatively more complex since dynamic websites contain a variable number of web pages and support CRUD operations. Moreover, if a website contains authentication, web crawlers won’t be able to access the web pages visible to users after logging in. In this case, a static sitemap generator will not produce accurate results. Even if a sitemap is generated this way, developers would have to constantly track and update the sitemap based on changes occurring in the website.

Nevertheless, we can generate a dynamic sitemap for a website in Laravel. For this, we require three things – a controller to define the logic for creating the sitemap, a route to define the route of the sitemap, and a kernel function to automate the process of updating the sitemap periodically.

Creating The Controller

To create a controller we use the following command in Artisan.

php artisan make:controller SitemapController

 

A controller is created in the ‘app’ directory under the ‘Controllers’ folder. Now, we need to define the command name and provide a description of the command. 

protected $signature = ‘sitemap:dynamic’;
protected $description = ‘Generate sitemap for the dynamic website’;

 

If you have any specific links that you would like to retain in the sitemap, you can declare them too. For that, you need to just define the base URL and the URLs of the websites that you would like to keep. Here, the base URL is typically denoted by ‘/’ but you can also use ‘app.url’ found in the .env file of your Laravel project.

$baseUrl = config(‘app.url’);


$sitemap = ‘<?xml version=”1.0″ encoding=”UTF-8″?>’ . PHP_EOL;
$sitemap .= ‘<urlset xmlns=”http://www.sitemaps.org/schemas/sitemap/0.9″>’ . PHP_EOL;

$fixedUrls = [                                   // define specific URLs
    $baseUrl,
    $baseUrl . ‘/login’,
    $baseUrl . ‘/password/reset’,
];

 

We can use this piece of code to loop through the fixed routes and generate the XML for the static links in the sitemap.

foreach ($fixedUrls as $url) 

{
    $lastmod = Carbon::now()->toIso8601String();
    $sitemap .= “\t<url>” . PHP_EOL;                            // <url> opening tag
    $sitemap .= “\t\t<loc>$url</loc>” . PHP_EOL;                // URL of the webpage
    $sitemap .= “\t\t<lastmod>$lastmod</lastmod>” . PHP_EOL;    // day last modified
    $sitemap .= “\t\t<priority>1.00</priority>” . PHP_EOL;      // priority of URL
    $sitemap .= “\t</url>” . PHP_EOL;                           // <url> closing tag
}

 

Now if we have a few URLs containing dynamic parameters, we can extract them from our database and automate the process of generating URLs for each parameter, similar to the example shown above.

$entityIds = Entity::pluck(‘id’);                        // extract all ‘id’s

foreach ($entityIds as $entityId)                        // loop through ‘id’s 
{
    $url = $baseUrl . “/entity/$entityId/holiday/index”;
    $lastmod = Carbon::now()->toIso8601String();

    $sitemap .= “\t<url>” . PHP_EOL;
    $sitemap .= “\t\t<loc>$url</loc>” . PHP_EOL;
    $sitemap .= “\t\t<lastmod>$lastmod</lastmod>” . PHP_EOL;
    $sitemap .= “\t\t<priority>0.80</priority>” . PHP_EOL;
    $sitemap .= “\t</url>” . PHP_EOL;
}

$sitemap .= ‘</urlset>’ . PHP_EOL;                       // close the <urlset> tag

 

Finally, we can save the sitemap generated and conclude the process.

$sitemapPath = public_path(‘sitemap.xml’);
File::put($sitemapPath, $sitemap);

$this->info(‘Sitemap generated successfully.’);
$this->info(“Sitemap file saved at: $sitemapPath”);
$this->info(‘GenerateSitemap command completed.’); 

 

So we should have the complete code as shown below.

<?php

namespace App\Console\Commands;

use Illuminate\Console\Command;
use Illuminate\Support\Facades\File;
use Carbon\Carbon;
use App\Models\Entity;

class GenerateSitemap extends Command
{
    protected $signature = ‘sitemap:dynamic’;
    protected $description = ‘Generate sitemap for the dynamic website’;

    public function handle()
    {
        $this->info(‘GenerateSitemap command is running…’);
        $baseUrl = config(‘app.url’);

        $sitemap = ‘<?xml version=”1.0″ encoding=”UTF-8″?>’ . PHP_EOL;
        $sitemap .= ‘<urlset xmlns=”http://www.sitemaps.org/schemas/sitemap/0.9″>’ . PHP_EOL;

        $fixedUrls = [
            $baseUrl,
            $baseUrl . ‘/login’,
            $baseUrl . ‘/password/reset’,
        ];

        foreach ($fixedUrls as $url) {
            $lastmod = Carbon::now()->toIso8601String();

            $sitemap .= “\t<url>” . PHP_EOL;
            $sitemap .= “\t\t<loc>$url</loc>” . PHP_EOL;
            $sitemap .= “\t\t<lastmod>$lastmod</lastmod>” . PHP_EOL;
            $sitemap .= “\t\t<priority>1.00</priority>” . PHP_EOL;
            $sitemap .= “\t</url>” . PHP_EOL;
        }

        $entityIds = Entity::pluck(‘id’);

        foreach ($entityIds as $entityId)
        {
            $url = $baseUrl . “/entity/$entityId/holiday/index”;
            $lastmod = Carbon::now()->toIso8601String();

            $sitemap .= “\t<url>” . PHP_EOL;
            $sitemap .= “\t\t<loc>$url</loc>” . PHP_EOL;
            $sitemap .= “\t\t<lastmod>$lastmod</lastmod>” . PHP_EOL;
            $sitemap .= “\t\t<priority>0.80</priority>” . PHP_EOL;
            $sitemap .= “\t</url>” . PHP_EOL;
        }

        $sitemap .= ‘</urlset>’ . PHP_EOL;

        $sitemapPath = public_path(‘sitemap.xml’);
        File::put($sitemapPath, $sitemap);

        $this->info(‘Sitemap generated successfully.’);
        $this->info(“Sitemap file saved at: $sitemapPath”);
        $this->info(‘GenerateSitemap command completed.’);
    }
}

 

Defining The Route

We can define the route of the sitemap in the ‘web.php’ file of our project.

Route::get(‘/sitemap.xml’, function() 

{
    return \Illuminate\Support\Facades\Redirect::to(‘sitemap.xml’);
});

 

Defining The Schedule Method In The Kernel

The script can be triggered automatically to run at a particular time. 

use App\Console\Commands\GenerateSitemap;             // define the controller path

 

protected function schedule(Schedule $schedule)
{
    // Run the command daily at 5 PM
    $schedule->command(GenerateSitemap::class)->dailyAt(’17:00′);
}

 

References:

  1. https://www.uplers.com/blog/all-about-of-sitemap/
  2. https://www.linkedin.com/pulse/understanding-difference-between-dynamic-static-sitemaps-arslan-elahi
Holiday API!

Holiday API!

Holiday API!

Save time in maintaining holiday data in multiple systems. We are already maintaining the data for NSE INDIA can add other organizations. You simply get API keys from us and start using it and save time. Our custom-made and easy-to-use Holiday API helps you check whether a organization has the day off on a particular day. Developers can also integrate our RESTful Holiday API with their custom applications to help users check holidays with ease.

Please contact us to get your email ID set up in this portal. You will receive an API key from our team.

https://holidays.atilagu.com/

Access Holidays In One Place!

Customers, stay informed of the days off! With our tool, you can view the holidays of any organization in one place. Our tool makes sure to provide you with accurate information regarding organizations’ holidays to save your time. Our UI makes it easy for users to navigate through a wide range of organizations and their holidays.

 

Keep Holidays Up-To-Date!

Whether you are a regional or even a national level organization, our reliable holiday management tool can help you inform your customers of your days off. It’s quite simple, just share your holiday list with us and keep your customers updated with your holidays, ensuring smooth and hassle-free management.

 

Please contact us for more details: https://fairshare.tech/contact-details-of-software-company-pune/

Scraping Or Data Scraping or Web Scraping

Scraping Or Data Scraping or Web Scraping

Scraping refers to the process of automatically extracting information or data from websites and web pages. Scraping is done by building custom software or using software from the market. Ideal cost-effective scraping is when a software program runs automatically without human intervention and users. 

Scraping is not always possible from the beginning or could stop in between without notice. Sometimes, it is possible but not a fully automatic process and involves human intervention or monitoring. Here are the reasons:

  • Since scraping is not an official way to retrieve data, data source companies are not obligated to inform when they stop supplying the data. Missing data is the worst-case scenario; nobody other than the data source company can fix this. 
  • Another possibility is a change in data format, which will cause temporary issues and delays as the software developer will need to adjust the software code to match the new data format. 
  • A data source company might block the server IP addresses from where these scrapper programs run, making 100% automation difficult. In this case, scraping software needs to be run from laptops. This software mimics human interaction with the data source website. It opens a page, chooses values from a dropdown button, types text where needed, and clicks a button to retrieve data. This semi-automation also has risks since data source companies could add a CAPTCHA or user-id/password to stop even the semi-automation. 

If scraping carries so many risks, then why is it done?

Such data under consideration is often essential to running the business. The data source company knows it. They are the only source of the data. The data is public data, and it is available on their public domain website. Even in such cases, the data source company leaves no reasonable alternative but scraping from its website. It is as if they want you to scrape the data!

All search results